Building permit data: how to monitor permit portals at scale

Building permit data is public record in most U.S. jurisdictions — but it is published across thousands of independent county and city portals, each with its own fields, formats, and terms of access. Using it at scale is not a scraping problem; it is a reliability and data-quality problem.

Permits are among the most commercially valuable public records that exist: they announce funded construction activity with an address, a declared value, an applicant, and a date, weeks or months before that activity is visible any other way. Suppliers, subcontractors, lenders, insurers, and proptech products all want the same feed. This post covers what permit records actually contain, why the naive pipeline decays, and the architecture that keeps permit monitoring trustworthy.

What a permit record contains

• Permit number— the jurisdiction's identifier, and your best deduplication key.
• Permit type — commercial new construction, alteration, roofing, mechanical, and so on; the fastest relevance filter.
• Property — address and usually a parcel id, which links the permit to a place (and to other permits on the same site).
• Declared value — the stated construction value; imperfect, but the standard threshold field.
• Applicant and contractor — owner, developer, or contractor of record. On commercial filings, an empty or pending contractor field is itself a signal.
• Status and dates— filed, approved, issued, finaled, with timestamps that define the project's stage.

The fragmentation problem

There is no national permit feed. Each jurisdiction publishes its own records through whatever system it bought — a vendor portal, a custom search page, occasionally a clean open-data endpoint, sometimes PDFs of scanned forms. Field names differ, value formats differ, status vocabularies differ (“issued” in one county is “approved” in the next), and the same project can appear in two jurisdictions when it crosses a boundary. Any serious use of building permit datatherefore requires a normalization layer: one schema that every jurisdiction's records are mapped into, with the original payload preserved.

Why the naive pipeline rots

The first version of every in-house permit pipeline is a script per portal on a cron job. It works in week one, and then reality arrives:

• Silent breakage.Portals change markup, pagination, or vendors without notice. A scraper that isn't health-checked fails quietly, and the gap in your data is discovered weeks later — usually by a salesperson asking why a county went dark.
• Re-scraping everything. Without change detection, every run reprocesses every record, which is slow, hammers the source, and buries the two records that actually changed.
• Duplicates. The same permit arrives from a search page and a detail page, or from two portals, spelled two ways — and your counts inflate unless records resolve to one entity.
• No history.A spreadsheet row records one moment. When the value is amended or the status advances, yesterday's truth is overwritten instead of versioned.
• Permission drift. Nobody re-checks robots directives, access terms, or rate expectations after the first day — which is how well-meaning pipelines become abusive ones.

What reliable permit monitoring looks like

The fix is to treat every source as production infrastructure, not a script:

• Source registry with permission status. Every portal is a first-class source with explicit permission and license state, access notes, and a monitoring cadence. Collection refuses to run against anything not approved — enforced in the pipeline, not a wiki page.
• Checksum change detection. Each retrieved payload is hashed; unchanged content is recorded as a no-op instead of reprocessed. Monitoring cost scales with change, not with corpus size.
• Schema-mapped extraction with evidence. Fields map into one normalized schema while the raw payload, URL, and retrieval time are stored — so every value can be traced to its source record.
• Entity resolution. Permits resolve by permit number and parcel first, then normalized names — linking permits to properties and projects instead of piling up rows. (More on this in the entity-resolution post.)
• Versioned changes → typed events. A status moving from filed to approvedisn't an overwritten cell; it is a versioned change that emits a permit_approved event you can alert and act on.
• Source health as a metric. Every source carries last-success, last-failure, and a reliability score; a failing portal alerts you the day it breaks, not the month after.

Doing this with Ballast

Ballast runs exactly this architecture as a service: a governed source network with permission gates, checksum-deduplicated collection, schema-aware extraction with stored evidence, identifier-first entity resolution, versioned records, and typed permit events (permit_filed, permit_approved, contractor_identified) scored against your specification and delivered by dashboard, signed webhooks, and a versioned REST API. You define the counties, the value floor, and the fields; the monitoring is Ballast's job — including telling you, honestly, when a source is failing or awaiting permission review.