Coglin Mill Keeps Pace With BI, Data Warehousing Demands

Published: May 26, 2009

by Dan Burger

When IBM introduced the AS/400 in 1988, it was clearly ahead of its time with regard to its report and query capabilities. But business intelligence has advanced while IBM depended on Query/400 for too long. IBM stepped forward with DB2 Web Query, the replacement for Query/400, and ISVs such as Coglin Mill--with RODIN, its ETL software--have also stepped up to advance database technology.

The major enhancement in Release 6 of RODIN is the capability to capture data in real time. In the previous releases, the change capture functionality was limited to running in batch mode and most users scheduled it at the end of the day. The real time data change capture allows changes all day, everyday.

The data captures can be done from either local or remote journals, which is not new functionality for RODIN Release 6, but it's important to the capability of the product, particularly because remote systems are more frequently a part of business intelligence projects. Users can specify the tables that need to have changes captured and have those associated with the appropriate journal.

As those entries hit the journal receivers, they are picked up and put into capture tables. RODIN allows multiple types of capture. There are captures that include before and after images, and captures that record all inserts, deletes, and other changes.

RODIN is described as an extract, transform, and load (ETL) product that's used to feed data into a data warehouse or data mart. It is often used to sort through transactional data when there are huge volumes being processed each day, but is even more useful with non-transactional data.

Examples of transactional data include line items such as invoices, cash receipts, purchase orders--items that are dated or time stamped. Accessing changes is done by comparing run dates--comparing end-of-day today with end-of-day yesterday, for instance.

Non-transactional data are things like customers, products, and stores--data that is not associated with a point in time and seldom contains a transaction date or time stamp. However, there may be a time stamp noting something like a customer phone number changed on a specific date.

While discussing RODIN with Alan Jordan, vice president at Coglin Mill, he described a situation he recently experienced with a potential new client who has an extraordinary data table containing 1.5 billion rows. It is transactional data, so new records get written every day, but the data has no index, and because of the sheer volume of data that is churning through the system, the company doesn't really want to take on the task of creating an index.

"That's an example of where automated change data capture would be very advantageous on a transactional table," Jordan says. "Because if 10,000 rows get changed, then those 10,000 journal entries will get processed rather than the entire 1.5 billion. The benefit of real-time changes is that at the end of the day, when the data warehouse gets loaded, the data is already collected and doesn't require going to the journals to get it."

Although this scenario is applicable to a large enterprise, Jordan says new product enhancements and a modular pricing plan has brought BI and data warehousing to the small to mid size organizations as well.

This is primarily an ease of use issue.

Usually as performance increases and more functionality is added, any software product becomes more complex to use. RODIN Release 6 has improved the management of multiple data sources.

"Almost all of our customers want to pull some data from SQL Server databases and have data coming from other sources," Jordan says. "And it is almost always remote rather than local."

With this in mind, Coglin Mill designed RODIN to provide a consistent view and management features whether the user is working with SQL Server, Oracle, or DB2 tables.

IBM has shared information with some ISVs that enables their software to enhance DB2 Web Query. "In the case of Coglin Mill," says Rick Bause, Power Systems PR manager, "we shared with them how they could automatically populate DB2 Web Query metadata with their ETL tool called RODIN. Their tool is an IBM i-specific data warehouse builder, so as part of pulling data from the data sources and populating/loading a DB2 for an i-based data warehouse, they can also automatically popluate/refresh DB2 Web Query metadata. This is a nice piece of integration when using RODIN in a DB2 Web Query environment."

A Webinar that includes the RODIN integration with IBM DB2 Web Query is scheduled for June 2. Coglin Mill and several other IBM partners with DB2 products relating to business intelligence and data warehouses will be included. To register for this event, titled "Jump start implementations with DB2 Web Query ecosystem solutions," see www-03.ibm.com/systems/i/software/db2/webquery/db2_webinars.html.

Pricing options for RODIN Release 6 are based on the server for an enterprise-level application that includes all functions; or for smaller organizations that do not require full functionality, RODIN is priced on the number of modules purchased. For more information, visit the company's Web site at www.coglinmill.com.

RELATED STORIES

Coglin Mill Debuts Lower Cost Versions of ETL Tools

Data Warehouses: Know One When You See One?

IBM Updates DB2 Web Query Business Intelligence Tool

                     Post this story to del.icio.us
               Post this story to Digg
    Post this story to Slashdot