Data Warehouse - Persistent layer (Fundamental Layer | Operational Data Store - ODS)
A persistent layer is a data warehouse layer where data is persisted (ie never deleted).
A persistent layer is also known as:
- a fundamental layer
- an operational data store (ODS)
- an enterprise data warehouse (EDS)
To support operational reporting you need to store a lot of transitional information that changes frequently and has little historical or strategic value to the business. That type of stuff can significantly overburden the design and maintenance of a dimensional data warehouse, so, an ODS is used.
Implementing an ODS can significantly improve and simplify data capture for loading the data warehouse. Because you can control the design and implementation of the ODS, it can be closely integrated with the DW, such as sharing key dimensions and delta capture for fact loads.
- Actual Data: Mirror the source application structure
- History Data: Save eventually the modification of the source data in a scd format
- data quality layer
- Make the loading process of the Data Mart restartable
- Enable cross source business logic
- Can receive data directly in the good data type (Example from a web service)
- The loading phase can be split in two: first, the loading of source data in the persistent layer then the design of the data.
- Backup: As the whole system restartable is from this layer, you may backup only this layer.
- Bad data file (ie that doesn't represent the reality such as test file in production) can be suppressed easily in this layer.