How to verify scraped data before anyone acts on it

To verify scraped data, give every record an explicit verification state and a numeric confidence that only moves through recorded checks: machine validation of the extraction, corroboration across independent sources, and human review — with the raw payload kept as evidence so any field can be traced to its source.

Collected data has a credibility problem that clean UI makes worse: once a value is sitting in a neat table cell, it looksequally true whether it came from a structured government record this morning or a model's guess about a paragraph last month. If people act on the data — call the developer, reprice the SKU, file the bid — the cost of a wrong record is real. Verification is how you separate “we collected it” from “you can rely on it.”

Step zero: keep the evidence

Verification is impossible without provenance. Before any scoring, store for every record: the raw payload exactly as retrieved, the source URL, the retrieval timestamp, and a content checksum. This does three jobs at once: disputes become lookups instead of arguments; unchanged content can be skipped by checksum instead of reprocessed; and every extracted field has a receipt. A record that can't show its receipt shouldn't be presented as fact.

The verification ladder

Trust should be a ladder of explicit states, each earned by a recorded check — not a vibe. A workable ladder:

• unverified — collected and extracted, no checks yet. Visible, clearly labeled, never silently promoted.
• machine_verified — the extraction validated against the schema and passed sanity rules (plausible values, well-formed fields, no impossible dates or negative prices).
• multi_source_verified — at least two independent sources corroborate the record. Independence is the operative word: two pages syndicated from the same upstream feed are one source wearing two hats.
• human_reviewed — a person confirmed it. Reserved for what warrants the cost: high-value records, disputed ones, and samples that audit the machine checks.
• disputed / stale— the demotion states. A reviewer rejected it, sources contradict it, or it hasn't been observed recently enough to trust. Demotions matter as much as promotions; data that can only gain trust ends up all “verified” and none of it meaningful.

Confidence as an earned number

Alongside the state, keep a numeric confidence (0–1) with two rules. First, the starting value reflects the extraction method: a field mapped from structured source data starts high; a field pulled from free text by a model starts low. Second, every movement is a recorded event— which check ran, confidence before, confidence after. A score that can't explain its own history is decoration.

The human queue, used sparingly

Human review doesn't scale to every record — and doesn't need to. Route to people only what machines can't settle: records below the confidence threshold, records where sources disagree, and the occasional random sample to keep the automated checks honest. Each decision should raise or cut confidence sharply and be recorded like any other check, so the audit trail includes the humans too.

Verification decays

A record verified in March is not verified in November; the world moved. Track last_verified_at separately from last_observed_at, and let sustained silence from every source demote a record to stale rather than letting it coast on old credibility. Freshness is part of truth, not a nice-to-have.

How Ballast implements this

This ladder is shipped behavior in Ballast: raw payloads are stored with checksums and URLs, every record carries one of six verification states and a confidence that moves only through recorded checks, corroboration across independent sources upgrades records automatically, low-confidence and disputed items land in a review queue, and every record in the console and the API exposes its evidence — source, URL, retrieval time, raw record — one click away. Verified events even meter differently from unverified ones, because trust is the thing being sold.