SI Database / Journal

Clean Data Is Not Necessarily Correct Data

On this page

A customer record arrives with extra spaces around a name. A state appears as “ca” in one row and “CA” in another. One date says “Oct 4, 2026” while its neighbor uses “2026-10-05.” The problems are visible, and many can be repaired with simple rules. This is the comforting side of data cleaning: inconsistency seems to invite correction.

But an immaculate spreadsheet can still tell a false story. If a customer has moved, a perfectly formatted address may still be the wrong one. A transaction can have a valid date and the wrong amount. A duplicate may be a mistake or a legitimate second event. Cleanliness describes orderliness; quality depends on how well the information serves its purpose.

The limits of a formatting rule

Consider a field containing “03/04/2026.” Does it refer to March 4 or April 3? A system that chooses one interpretation without knowing the convention may replace an ambiguous value with a confident error. Similarly, automatically merging two records with the same name can erase two different people. The most responsible cleaning rule is sometimes not to change a value at all, but to flag it for review.

Our Data Cleanup lab shows two modest operations: normalizing a state abbreviation and removing unnecessary spaces, or turning a currency-like string into a number and standardizing a status label. The examples are curated to be unambiguous. They illustrate transformations; they do not prove that the resulting records match reality.

Six useful questions

Data-quality work is often organized around several dimensions: accuracy, completeness, consistency, timeliness, validity, and uniqueness. They sound similar, but they answer different questions. Is a value true? Is something missing? Do two sources agree? Is the information recent enough? Does it match the permitted format? Is a supposedly singular record present more than once?

A price of $87.00 can be valid in format and inaccurate in fact. A list of customer accounts can be complete but riddled with duplicates. A status field can be consistent across a database and consistently wrong. No single cleanliness score catches all of these problems.

What human judgment contributes

Before cleaning begins, someone needs to define which source is authoritative and which alterations are acceptable. Preserve the original value when a transformation might be contested. Keep a record of the rules applied. Test edge cases, including missing fields, unusual names, and dates whose format is uncertain. Where possible, compare the cleaned data against trusted external evidence, not simply against another version of itself.

Automation is extremely valuable for repeatable tasks. It becomes dangerous when its smoothness conceals an unsupported guess. The goal is not to make every field look tidy. The goal is to make information reliable enough to support a decision, and to make the limits of that reliability visible.

Sources and further reading

← Back to the journal