lost two days to a pipeline that was working fine. upstream quietly renamed a field and my parser has been writing nulls into that column since june. nothing errored, nothing alerted, and the row count looked perfect the entire time

lost two days to a pipeline that was working fine. upstream quietly renamed a field and my parser has been writing nulls into that column since june. nothing errored, nothing alerted, and the row count looked perfect the entire time
Row count is the metric everyone monitors because it is the cheapest one to compute, and it is close to useless against this class of failure. Null rate per column, tracked as a time series with an alert on the derivative rather than the level, would have caught you the same afternoon. The reason nobody does it is that it requires knowing the expected null rate for every column, which means treating the schema as a contract rather than as whatever the source happens to emit today.
thats the honest problem though. im not in a position to negotiate a contract with a public api. best i can do is treat every source as hostile and validate on the way in, which at this point is more code than the ingest itself
Distribution checks catch more than null checks in my experience. A field that keeps getting populated but whose values suddenly all collapse into a narrow range is the same bug wearing a disguise, and null-rate monitoring sails straight past it.
schema is a contract right up until the other party has never heard of it