Data lineage records the relationships between data sources, transformations and outputs for identified runs or versions. Quality checks test stated conditions, while reconciliation compares records across a defined boundary.

Origins, assertions and comparisons

Lineage answers where a result came from and which processing steps contributed to it. OpenLineage models datasets, jobs and runs, providing objects for those relationships.

A quality assertion tests a specified property. A column can be required to contain valid dates, an amount to use an accepted currency or a dataset to meet a defined count condition. Great Expectations represents such checks as verifiable assertions. Passing a check establishes the tested condition, not every property of the dataset.

Reconciliation compares independently maintained records or processing stages under an explicit mapping. It can identify missing events, amount differences or timing differences. Its conclusion depends on the compared population, units, cutoff and accepted exceptions.

A missing posting in a valid dataset

Take 100 source postings totaling 1,000 dollars and an extract containing 99 postings totaling 990 dollars. Every extracted field has the expected type, and the transformation has a recorded lineage path.

The valid types establish that the 99 records satisfy those type checks. The lineage establishes their recorded origin and processing relationships. Neither fact explains why one source posting is absent. Comparing source and extract counts identifies a one-record gap; comparing their amounts identifies a 10-dollar gap.

The reconciliation then needs identifiers to determine which posting is missing. Equal totals alone would not establish equal populations: an omission and a duplicate could offset numerically. Counts, amounts and record relationships answer different completeness questions.

Pipeline runs and corrected data

Change-data capture transfers source changes through processing and transport stages. Each stage has a boundary at which a record can be delayed, replayed or transformed. A run identifier and source processing position tie a quality or reconciliation result to a particular output.

A successful scheduler task establishes that the task reached its configured success condition. If that condition is merely completion of execution, it says less than a result tied to reconciled source coverage.

Manual adjustments and external corrections also belong in the data history. A lineage graph that records only automated jobs can omit transformations that change the financial result.

Scope of data-control evidence

What changes across datasets is the required population, transformation and business rule. The distinct outputs remain origin, tested property and explained difference.

Reconciliation does not prove that both compared records are economically correct. Two systems can agree on the same erroneous input. The comparison’s boundary and the quality of the underlying evidence therefore remain part of its conclusion.

Questions about data lineage

Does complete lineage prove data accuracy?

No. A recorded transformation can preserve an incorrect value or omit a record.

Does passing quality checks prove every source record arrived?

No. Completeness requires a check against the intended source population.