(Data Processing|Data Integration)

1 - About

Data processing is a more general term for manipulating data whereas data integration is the integration of data between two systems.

3 - Management

3.1 - Model

Data Integration has roughly two data processing model:

stream processing Send a message to another process, to be handled asynchronously
Processing that executes continuously as long as data is being produced
batch processing Periodically crunch a large amount of accumulated data
Processing that is executed and runs to completeness in a finite amount of time, releasing computing resources when finished

3.2 - I/O Pattern

Application Query Selectivity Processing Data
OLAP Changing Queries (ad-hoc) A lot A lot Fixed Data
OLTP Fixed Queries Few Few Changing Data
Streaming Fixed Queries All Few Changing Data
Batch (Data Warehouse) Fixed Queries All A lot Fixed Data

see also I/O - Workload (Access Pattern)

3.3 - Data Processing Model / Framework

3.4 - Function

See operations.

A woman using a keypunch to tabulate the United States Census, circa 1940:

4 - Others

4.1 - Goal

  • The “360 degree view of the enterprise” is a commonly discussed goal that really means data integration. ??

4.2 - Term

  • ETL : Extraction, Transformation and Load Software
  • ELT : Extraction, Load and Transformation Software

4.3 - Magic Quadrant

5 - Documentation / Reference

Data Science
Data Analysis
Data Science
Linear Algebra Mathematics

Powered by ComboStrap