A vendor's daily file keeps the same column names and types, but a column that was never null starts arriving null on 2 percent of rows and a categorical column gains a value never seen before. Nothing fails. Give the monitor that catches both on the day they happen.

A vendor's daily file keeps the same column names and types, but a column that was never null starts arriving null on 2 percent of rows and a categorical column gains a value never seen before. Nothing fails. Give the monitor that catches both on the day they happen.

Approach: Monitor the distribution of the data rather than the declared schema, choosing per column statistics whose normal range can be learned from previous loads.

Profile every column on every load and compare against a trailing baseline window: alert when a column's null rate leaves its historical band, and when a categorical column's set of distinct values gains a member that has never appeared. A type check passes here because the declared types did not change, which is what makes this silent schema drift rather than a load failure. The per column profile is cheap and fixed: row count, null rate, distinct count, min and max for numerics, and the top values with their shares for low cardinality columns. A column with a null rate of 0.0 percent across 200 loads that arrives at 2.0 percent is a hard alert rather than a soft one, because its historical band has zero width. Cardinality drift needs the value set stored rather than only the count, since a new value can replace a retired one and leave the count unchanged. Two details make the monitor usable: compare against the same weekday or a trailing 20 loads, because a Monday file legitimately differs from a Friday one, and route the alert to a quarantine decision rather than a page, since a new categorical value is usually a legitimate new venue code and the pipeline must refuse to map an unknown value onto a default.

Follow-up: How do you tune the null rate band for a column whose null rate genuinely varies with market activity?

Key concepts: null rate monitoring, cardinality drift, schema drift, baseline window.