[A companion to the Demystifying Data Governance series]

Expectations, Baselines, and What We Actually Guarantee

Imagine this: a number is wrong, but no one can find why. An analyst notices the issue and reruns the query, but it returns the same value. An engineer checks the pipeline and finds every run green: the freshness check passed, the row counts are in range, and the schema tests are clean. A product manager sees a metric that moved, and no one can explain why.

The situation is similar to writing software: it is compiled, but the result is incorrect. So, we write unit tests to ensure that a function always works as expected. In the data world, that’s the role quality checks play.

The analogy holds until it does not. A unit test is written against a specification – someone decided what the function should return before the test existed. However, most data quality checks have no specification. They assert what someone assumed, or what the data looked like when the check was written. That is why the pipeline can be green but the number still wrong: the checks answered the questions someone thought to ask, but the failure happened elsewhere.

Most writing about data quality starts from a list of dimensions, such as accuracy, completeness, timeliness, consistency, uniqueness, and validity. That list is useful, but it does not explain the failure above. Anyone who has read the list still gets ambushed by it.

This article approaches data quality from a different angle: Data quality is not a property data has or lacks. It is the gap between what we wrote down as an expectation and what the data actually is. Everything practical follows from that: what to profile, what to guarantee, where to put checks, what to spend, and why so many failures pass through a full test suite without a single alert.

Quality Is a Gap, Not a Property

Normally, when we ask what makes a data set high quality, the answers are simply that it is accurate, complete, or fresh. But none of these means anything until someone says what the data is for. For example, a customer table with a 4% null rate on the phone_number column is high quality for a team building a monthly revenue report and unusable for a team building an SMS notification service. The same data, but with two expectations.

So, there are two sides, and quality lives between them.

Observation is what profiling gives us. A profile is a statistical summary of a dataset, including cardinality, null rate, distribution, minimum and maximum, and outliers. Profiling is the first step for any dataset we intend to trust. However, it comes with a cost. A profile is a full scan, and some of the most useful statistics, such as exact distinct counts, quantiles, and cross-field relationships, need sorting or multiple passes. On a large table, it is expensive, and the cost recurs every time we want a fresh picture.

Besides, profiling is also frequently misread. A profile describes the data’s current state; it is neither a target nor a verdict. If a field is 30% null, the profile has no opinion about whether that is a disaster or completely expected. Therefore, people who treat the profile as the answer end up monitoring numbers without knowing which direction to go.

Expectation is the other side, and it is usually neglected. An expectation is a written statement about what the data must be for a specific use case defined by its users. It has to be well defined and specific. For example, the notification service cannot send to a null phone number, so every row flagged sms_opt_in = true must have a non-null phone number. Although that is a longer sentence than “completeness,” it is worth the length because it is testable and names who cares.

Those six words from the opening — accuracy, completeness, consistency, timeliness, uniqueness, validity — are the standard dimensions of data quality. Each names a kind of gap:

They are useful as labels. The SMS expectation above is one sentence, but it covers two dimensions: completeness (the number must be present) and validity (it must be E.164). Labels like these help group checks and report on them.

What the labels do not do is set the bar. “Completeness” does not say whether 4% null is acceptable — the revenue report says yes, the SMS service says no. Only an expectation says that. This is why a team can know all six dimensions and still get caught out: knowing the categories of failure is not the same as having written down what counts as one.

So, a dataset with no written expectation has no quality state at all. Not good, not bad, but undefined. We can profile it and chart it and still not say whether any value is a problem, because there is nothing to compare it with. Much of what gets called a data quality problem is exactly this: nobody wrote down what the data had to be, and someone downstream found out the hard way.

When Nobody Knows What to Expect

Ideally, the consumer tells us what to expect since they know what breaks their work. Unfortunately, this isn’t always the case, especially when ingesting a new source (e.g., a new vendor feed or a new application’s events) before anyone has built on it. Consumers also don’t know what to expect yet; they haven’t used the data long enough to know what normal looks like. So, instead of defining expectations first, we pave the way to achieve them.

• Know. As the first pillar of the series (What Do We Have and What Does It Mean?) mentioned, we cannot check something we don’t know. When no one knows the expectation, profile the data and learn what it actually contains, which fields are populated, what values appear, and how volumes move day to day. Although it sets no expectations, it’s the foundation for building them.
• Baseline. Document what we learned and alert when something looks anomalous: the null rate jumps, the row count leaves its usual band, or a field’s distribution shifts. This doesn’t require knowing what’s correct, only history, so it can start the week the profile exists. Like any other check, it belongs to the fields that matter, not every column we happen to have.
• Stated. As consumers build on data, they learn what breaks their work, and that is where the stated expectation comes from. When a consumer can say, “below this, my numbers are wrong”, the baseline check gets promoted to a stated bar, with a consumer, an owner, and a reason. Baseline checks that nobody ever promotes deserve a question too: a change no one would act on is not worth an alert.

The three steps pave the way from not knowing to knowing: first what the data is, then when and what changes, then what has to be. Only at the last step do we know enough to promise anything about it. That is when a quality check can become a guarantee – one that stops bad data instead of just reporting it.

Keep in mind that the stated expectation isn’t permanent. It is an agreement that holds for as long as its reason does. Consumers change what they need, sources change what they send, and a check that has not fired in a year may be guarding something nobody uses anymore. So, the path is a loop: keep monitoring, review each expectation against its reason, and go back to knowing the data when the answer has changed. That is why every expectation needs a written reason. If nobody can explain an expectation, we can’t re-evaluate it; we can only keep it or delete it on instinct.

Where Expectations and Checks Live

An expectation is knowledge, so we naturally keep it in a document and write quality checks in the code. However, like any other software documents and code, eventually, they drift. As software engineers, we all know the DRY principle: “Don’t Repeat Yourself.” So we should apply the same principle to expectations, keep a single source of truth, and generate the quality checks from it, or alongside it.

One idea is to document expectations as configuration files (e.g., a YAML file) and keep them next to the pipeline code. Whoever defines the expectation writes it there, and whoever needs to review it reads it there. The checks are generated from the same file or kept right beside it in the repository, so a change to one is a change to the other.

The following is an example of the configuration file (a YAML file).

- name: sms_phone_present
  column: phone_number
  statement: >
    Every row with sms_opt_in = true has a non-null,
    E.164-formatted phone_number.
  consumer: notification-service
  owner: customer-data-team
  reason: The SMS service cannot send to a missing or malformed number.
  type: stated
  check:
    where: sms_opt_in = true
    not_null: true
    pattern: '^\+[1-9]\d{1,14}$'
  on_failure: drop_and_count

This is not a new idea. Most quality tools, such as dbt in its YAML, already keep a check and its description in the same file, next to the code. However, they usually leave out the two fields that turn a check into an expectation: who it is for and why it exists. Those two fields answer the questions a check alone cannot – who to call when it fails, and whether it still matters at the next review. The complete version, with both, is what the industry calls a data contract.

Where Should a Check Run

Although a single copy of expectations is the source of truth, where checks run is a separate decision, since checks can run anywhere in a pipeline. A check’s return value determines where it should run, because checks aren’t free. The common places are the following.

• Where a promise comes in. If a vendor or an upstream team guarantees something, e.g., a file by midnight without missing account IDs, check it the moment the data lands. That catches the problem as early as possible, when it is cheapest to fix, and it holds the promiser to the promise.
• Where the data changes. Any transformation can introduce a failure that did not exist upstream – a join that fans out, a filter that drops too much, a calculation that is simply wrong – and no upstream check can see it. Check right after the transformations whose output someone depends on.
• Where the data leaves. A report to a regulator, a number on a customer’s statement, a file to a payment processor. Once it is out, it cannot be quietly fixed, so check before it goes.

The check most often missed is reconciliation at the end. Structural and validity checks are easy to write generically, so they’re usually included. Reconciliation—checking whether totals match a known figure or this month agrees with the prior close—requires understanding what the numbers mean, so it’s often skipped. These are exactly the failures that make it onto a dashboard and get noticed by a person.

Where a check sits determines what happens when it fails. At a promise or exit, it should stop the data—by blocking the load, quarantining the batch, or holding the report. This is a guarantee. A check that only reports is a dashboard: useful for investigation, but it protects no one.

Stopping bad data at the source is a basic quality principle: preventing a defect costs less than fixing it. A bad row caught at the boundary requires quarantine. If we find it a week later, after feeding dashboards, retraining a model, and producing a board report, it may require a backfill, a restatement, and lost trust. Labovitz and Chang’s 1-10-100 rule in Making Quality Work estimates that preventing a defect costs about $1, correcting it during processing costs $10, and fixing it after it reaches the customer costs $100. These are rough figures, but the principle holds: the cheapest fix is preventing the bad row from being created.

Follow the Lineage

Lineage turns these spots into specific tables: which ones deserve a check, and where it should sit in the chain.

Check for failures at the highest point possible. If one source table feeds twelve downstream tables, a single null check on the source replaces twelve downstream checks—less to run and maintain. Teams often add duplicate checks because each seems inexpensive, but lineage reveals they’re the same. The highest point is the producer: enforce the rule when data is written, before it leaves the application. Catch failures introduced by a transformation where they occur.

Count and weigh the consumers. Lineage shows how important a dataset is before anything goes wrong. A table used by one exploratory notebook carries less risk than one powering a revenue dashboard, a regulatory report, and two ML models. The lineage graph makes that difference visible. Count the consumers, then weigh them: one regulatory report matters more than ten notebooks. This is the same number as the blast-radius metric discussed later, measured before an incident rather than after.

This is the most defensible way to decide which datasets need thorough checks, but it depends on complete lineage. If the graph is incomplete, it will undercount consumers and make important tables appear less critical.

Check Sensitive Columns Where the Values Are Visible

In the access control article, we discussed how sensitive columns are often hashed, masked, or encrypted before analysts see them. This can lead to too few checks, even though sensitivity usually signals importance. But checks on masked values test only the mask, not the underlying data; especially with format-preserving encryption, ciphertext is designed to look valid, so every format check passes. Such results create false confidence and should not count toward coverage.

Run checks before masking or use a quality job authorized to read plaintext. Either way, access should be deliberate, scoped, and audited. Classification—the output of discovery—identifies protected columns. Without it, a quality program cannot know where its checks are blind.

What Checks Cost

Running quality checks has a cost. Each check has two costs: compute and latency.

Compute is the obvious cost, and usually the smaller one. Scanning a large table for a distribution shift costs money, but it shows up on a bill and can often be reduced with sampling.

Latency is less obvious but often more significant. A blocking check delays the data before it can be used. A ten-minute reconciliation before publishing adds ten minutes to every downstream consumer’s wait, every time it runs. Non-blocking checks avoid that delay but let bad data spread. This is the same speed-versus-safety trade-off discussed in the access control article, appearing here in a different pillar.

A check is worth running when its cost is lower than the expected cost of the failure it catches: the impact of the failure multiplied by how often it occurs. Cheap checks on valuable data are worthwhile. Expensive checks for rare, low-impact failures may not be.

Profiling follows the same rule. A full profile is essentially another check: it scans the same data and uses the same compute. Profile thoroughly only when lineage shows that the data’s downstream importance justifies the cost.

Uneven coverage, then, can be a deliberate design choice—not a gap. Thorough checks on twelve critical datasets and few checks on four hundred low-traffic tables may be the right allocation. The key is whether that allocation was intentional or simply the default.

What to Measure About Our Checks

We use quality checks to assess our data, but we also need metrics to ensure those checks work. The following metrics reveal more about a quality program than a dashboard full of green ticks.

• Detection source ratio: What share of quality incidents are found through monitoring rather than reported by people? This measure shows whether our checks focus on what matters. If analysts report most incidents, our checks test the wrong things, however many checks pass. Record this measure before adding checks to establish a baseline and show whether the program makes a difference.
• Time to detect: The time between bad data arriving and someone noticing. Expected failures are detected within hours; unexpected ones can take days or weeks. The gap between these two is what matters.
• Blast radius: How many downstream tables, dashboards, and models consumed the bad data before detection? This translates a quality incident into a number leadership already understands.
• Expectation coverage: Measure the fraction of columns people rely on that have a written expectation: what each column must contain, for whom, and why. This differs from having checks. A column may have several generic checks, such as not-null or unique, without anyone documenting what it should contain.
•  Threshold decay: What fraction of checks were widened, muted, or disabled without review last quarter? This reveals checks being quietly weakened—a practice almost nobody tracks. A check tightened after an incident and quietly widened six months later is a failure waiting to happen.
• Orphans: Checks with no written expectation behind them are rules nobody can explain. Expectations with no check enforcing them are promises nobody keeps. Both should trend toward zero, and a rising count in either is drift showing up before an incident does.

A Worked Case: Updated Hourly, but Hours Behind

A freshness check can answer two different questions:

• Did the job run? Did it start on time, finish without errors, and write today’s partition?
• Is the data current? How old is the newest row?

They sound alike, but they’re not. A job can run on schedule, succeed, and write to the table every hour while its data grows increasingly stale. Checks of run status, runtime, or the table’s last-modified time still report it as fresh. Only a check comparing the newest row with the current time catches the delay.

Most monitoring answers only the first question because the tools make it easy. The same distinction applies to SLAs: “The ETL completes by 6 a.m.” promises the job ran. “The data reflects transactions through midnight by 6 a.m.” promises the data is current. Only the second promise is useful to consumers—and difficult to meet.

The case below is this second kind of failure. It slipped past every check.

What Was Expected

An hourly pipeline copies new records from PostgreSQL to a data lake table. Each run reads records created during the previous hour; for example, the 10:00 run reads records created between 9:00 and 10:00. Consumers assumed the table was current within an hour or two because the pipeline ran hourly.

What Was Observed

Runs began taking over an hour. Because each run waits for the previous one to finish, they started late but still read their scheduled hour. Nothing was skipped or duplicated; the data simply fell further behind each time.

At 70 minutes per run, the pipeline falls 10 minutes behind with every run. After a day, the table is about four hours behind. After a week, it is more than a day behind, even though it is still updated every 70 minutes. It does not catch up on its own.

Why Every Check Passed

Every check asked a question about the job, and every answer was true.

• Job status. Every run succeeded. No errors, no retry.
• Table history. The metadata showed a new write about every hour. A “last updated within two hours” check passed every time.
• Volume. Each run loaded one normal hour of records, so row counts looked like any other day.
• Content checks on each batch. Nulls, schema, and duplicates were all fine because each batch matched its hour.

Even a newest-timestamp check passes when it compares against the run’s scheduled hour rather than the current time. Each run was correct by its own clock; the clock itself drifted.

What Would Have Caught It

The check measures data age: the time since the newest business timestamp in the target table. In other words, how old is the newest row? Teams that use the table agree on a threshold in advance, such as: “The newest row should be no more than two hours old.” The check runs on its own schedule, not as part of the pipeline. A check that runs after each write can’t detect when writing stops. This check would have alerted within hours. None of the four checks above did.

- name: orders_data_age
  column: created_at
  statement: >
    The newest row is no more than two hours old,
    measured against the current time.
  consumer: <consumer>
  owner: <owning-team>
  reason: <what breaks when the data is stale>
  type: stated
  check:
    max_age: 2h
  on_failure: alert

Two pipeline metrics would have warned even earlier, before any consumer noticed:

• Run duration ÷ schedule interval. Above 1, the pipeline cannot keep up, even while every run is green.
• Start delay. The actual start minus the scheduled start. If it grows run after run, a backlog is forming.

The data-age check is the guarantee because it states what consumers need. The two pipeline metrics are dashboards that explain why the guarantee is about to fail.

What It Cost

Without the check. The table was several hours behind when a downstream team’s daily job failed: it looked for the new day’s data, but it wasn’t there yet. That much lag took days of green runs to build.

• Detection source: a consumer, not our monitoring.
• Time to detect: days from the first slow run.
• Blast radius: one downstream job failed outright. Every other report and query that read the table in those days used stale data without knowing it.

The failed job was the cheapest part, because at least someone noticed it. The stale reads that never failed are the cost nobody can count.

With the check

A scheduled query checks the newest timestamp in a table. If the column is indexed or partitioned by time, it reads only a few rows—not the whole table. The compute cost is negligible, and the check runs alongside the pipeline without slowing it down. Writing the expectation takes minutes. It would have caught the first slow run within two hours, days before the downstream team noticed.

Days of stale data, versus one scheduled query.

Where to Start

Most of the time, we don’t start from scratch. Hundreds of tables already exist, and most have no checks. The question is not how to design a quality program but where to begin. This section outlines the steps to get started.

1. Rank before profiling. Start with the inventory, not the data. Identify which datasets exist and which are actually used. Lineage and query logs answer these questions without scanning rows; begin with the inventory questions in the self-evaluation checklist. Rank datasets by the number of consumers and the importance of what depends on them, as described in Follow the Lineage. The result is a short list of the dozen datasets whose silent failures would be most costly. The same review may uncover unused tables—candidates for cleanup, not quality checks.

2. Profile the short list. This is where it’s worth paying for the more expensive step: profile only the shortlisted datasets to find out what they contain.

3. Ask the consumers, then check. Existing data has one advantage over new data: consumers have already lived with it. They know what disrupts their work because they’ve experienced it firsthand. Their past complaints are a starting point for defining expectations. Record those expectations, then use baseline checks to cover the rest of the short list.

Tell consumers before turning anything on. A new check on an old table immediately flags defects that may have existed for years. Start each check as a dashboard, then make it a guarantee once it has proven itself. Otherwise, the program’s first impression will be a wall of red.

Closing: Quality Sends Us Back to Discovery

The four pillars of the series read like a progression — know the data, secure it, use it properly, improve it. In practice, the fourth one does not sit at the end of a line. It runs back to the beginning.

We cannot set expectations for data we do not understand. Every effort to improve quality sends us back to basic questions: What is this data? Who uses it? What do they need from it? That is discovery, approached from the other direction.

This is not a failure of sequence; it is part of the work. Quality efforts return us to the catalog with a sharper question: What must this column mean for downstream users to trust it? The answer gives us an expectation we can test.

Seen this way, the four pillars are promises: Know the data—we can explain what we have. Secure the data—we can say who accessed it. Use data properly—we can explain why we used it. Improve quality—we can say whether it can be trusted. Each promise has someone waiting for an answer. That is what distinguishes governance from engineering. Making a pipeline faster is a technique. Promising that data reflects transactions through midnight is a commitment—with an owner, a scope, and a way to verify it was kept.

This article is a companion to the Demystifying Data Governance series.

• The series article this expands on: How Do We Know It’s Working?
• The prerequisite: What Do We Have and What Does It Mean?
• Self-evaluation: Do We Really Know Our Data? A Self-Evaluation Checklist