The goal of Change Data Capture is to track change in the source data. When running integration interface, ODI-EE can reduce the volume of source data processed in the flow by extracting only the changed data.
Reducing the volume of source data is useful in many field such as:
These changes are captured by Oracle Data Integrator and transformed into events that are propagated throughout the information system.
Changes tracked by Changed Data Capture constitute data events. The ability to track these events and process them regularly in batches or in real time is key to the success of an event-driven integration architecture.
Changed Data Capture is performed by journalizing models. Journalizing a model consists of setting up the infrastructure to capture the changes (inserts, updates and deletes) made to the records of this model's datastores.
Oracle Data Integrator supports two journalizing modes:
Changed Data Capture uses a publish-and-subscribe model. This model works in three steps:
ODI-EE processes datastore changes in two ways:
The journalizing components are:
These components are implemented in the journalizing infrastructure.
Simple Journalizing enables you to journalize one or more datastores. Each journalized datastore is treated separately when capturing the changes.
This approach has a limitation, illustrated in the following example: Say you need to process changes in the ORDER and ORDER_LINE datastores (with a referential integrity constraint based on the fact that an ORDER_LINE record should have an associated ORDER record). If you have captured insertions into ORDER_LINE, you have no guarantee that the associated new records in ORDERS have also been captured. Processing ORDER_LINE records with no associated ORDER records may cause referential constraint violations in the integration process.
Consistent Set Journalizing provides the guarantee that when you have an ORDER_LINE change captured, the associated ORDER change has been also captured, and vice versa. Note that consistent set journalizing guarantees the consistency of the captured changes. The set of available changes for which consistency is guaranteed is called the Consistency Window. Changes in this window should be processed in the correct sequence (ORDER followed by ORDER_LINE) by designing and sequencing integration interfaces into packages.
Although consistent set journalizing is more powerful, it is also more difficult to set up. It should be used when referential integrity constraints need to be ensured when capturing the data changes. For performance reasons, consistent set journalizing is also recommended when a large number of subscribers are required.
ODI-EE provides two methods for tracking changes from source datastores to the Changed Data Capture framework:
The triggers method creates triggers on the source tables to track changes as data is inserted, updated, or deleted. This method can be implemented on most RDBMS, but it can have an impact on the transactional performance of the source systems.
The second method involves mining the RDBMS logs, which are the internal change history of the database engine. This method has no effect on the system’s transactional performance; it is database-specific. This method is supported out-of-the-box for:
The Changed Data Capture framework used to manage changes is generic and open. The change tracking method can be customized, and any third-party change provider can be used to load the framework with changes.
Developers define the declarative rules for the captured changes within the integration processes in the ODI-EE Designer graphical user interface—without having to code. With the Designer, customers declaratively specify set-based maps between sources and targets, and then the system automatically generates the data flow from the set-based maps. The technical processes required for processing the changes captured are implemented in Knowledge Modules.
Changes frequently involve several datastores at one time. For example, when an order is created, updated, or deleted, it involves both the orders table and the order lines table. When processing a new order line, the new order to which this line is related must be taken into account.
ODI provides a mode of tracking changes, called Consistent Set Changed Data Capture, for this purpose. This mode allows you to process sets of changes that guarantee data consistency.