Method
How a compilation is built, and what gets refused along the way
Nothing here is proprietary in an interesting way. It is ordinary data work done in a fixed order, with the rules written down so a buyer can argue with them and reproduce the parts that matter.
§01 Choosing a source
A dataset counts only once code has read it
Plenty of government datasets look promising in a search result. A source is only treated as usable here when working code retrieves it, its record-selection rules are written and tested, and the date of that verification is recorded. Anything short of that stays on the coverage page under "not offered", with its identifier named so the exclusion is visible.
- Only sources published for public use are retrieved, on the terms the publisher sets.
- Robots directives and page terms are read before a page is requested.
- Where an agency answers a records request instead of publishing a dataset, the request and its response are kept with the extract.
§02 Retrieval
Every pull is stamped twice
Each build records the date it pulled, and separately the timestamp the agency itself last updated that dataset. Both are printed on the first tab of the workbook and written into the release manifest. That is what makes a figure on one of these sheets checkable rather than merely confident.
| Dataset | Identifier | Agency last updated |
|---|---|---|
| DOB Safety Violations | 855j-jady |
2026-06-26 18:08 UTC |
| MapPLUTO | 64uk-42ks |
2026-05-28 19:50 UTC |
Scroll the table sideways for the remaining columns.
§03 Matching
Attaching the building to the record
A violation record on its own says little about the building. Rows are joined to the city's tax-lot dataset by borough-block-lot to attach building class, floor count, unit count, owner of record and community district. The join is a numeric key, not a fuzzy address match, and the rate it achieves is measured on every build and gated.
- Join rate must reach 0.90. On the 2026-06 build it reached 0.996.
- Building and lot identifier coverage must reach 0.98. On that build it reached 1.000.
- A row that cannot be matched is not quietly filled in from a similar address.
§04 Qualifying
The agency's exemption rules, not an approximation of them
Eligibility is decided by the published rule, checked verbatim against the agency's own guidance and dated. Where the rule turns on something public data does not carry, the row is kept and flagged rather than dropped or assumed.
Worked example: gas piping eligibility
- Classes on the published exempt list are dropped outright.
- Class B0 is absent from that list, so it is kept. It would be easy to drop it by resemblance to neighbouring classes; the list governs.
- Class M3 is exempt at two dwelling units or fewer. Dropped only where the public unit count proves it, kept and flagged otherwise.
- Class M4 turns on occupant count, which public data does not publish. Kept and flagged, never guessed.
Commercial filtering
Where a compilation is restricted to commercial and multifamily property, the filter runs on building class, because class is the authoritative fact. Filtering on whether an owner's name looks like a company is unreliable in both directions: a person can own a hotel, and a limited company can own a two-family house.
On the 2026-06 gas build this filter was off, and the count reflects that.
§05 Ranking
Order you can argue with
The working queue is ordered by cycle year due, then building size by floors and units,
then how long the record has been open. Every row carries a rank_reason
written in words, so the ordering can be checked instead of taken on faith. If the reason
looks wrong for your trade, say so and the ordering can change.
§06 The registry
What changed since last time
Every building ever seen active in a territory is kept in a registry, so each build can say whether a row is new since the last pull, recurring from an earlier one, or part of a first baseline. Rows that were active before and are not active now go to their own tab rather than disappearing, because a record that resolved is information too.
- New — newly delinquent since the previous pull.
- Recurring — open in a prior pull and still open.
- Baseline — the first pull for a territory, with nothing to compare against.
- Cured since prior — active last time, not active now.
§07 Returned outcomes
Dead rows stop coming back
Each workbook ships with a blank return sheet. Send it back and the next build suppresses what it should. Six dispositions take a row out of the queue: won, already cured, no gas service, already has a vendor, bad record, and do not contact. A row marked lost is recorded but deliberately not suppressed, because a job lost this cycle is a job to re-offer next cycle.
Suppressed rows stay in the master tab and the registry with the reason attached. Nothing is silently deleted.
§08 Release checks
Ten gates, nine of them able to stop the build
The checks run after the work and before the file. A hard gate failing does not produce a warning banner on a file that ships anyway; it stops the release and nothing is written.
- Column schema complete; queue not empty; no duplicate keys.
- Join rate and identifier coverage above their thresholds.
- Zero rows without a real record number, because a row without one cannot be verified and should not be acted on.
- Zero rows from an exempt class leaking through the eligibility rules.
- Every source carrying its update timestamp.
- Active-record count within 40% of the prior pull, so a source outage shows up as a blocked build rather than a thin file.
- One soft check on the mix of record types, which warns rather than blocks.
| Check | Threshold | Measured | Result |
|---|---|---|---|
schema_columns |
No missing columns | All present | PASS |
queue_nonempty |
> 0 | 250 | PASS |
pluto_join_rate |
≥ 0.90 | 0.996 | PASS |
bin_bbl_coverage |
≥ 0.98 | 1.000 | PASS |
violation_evidence |
Zero rows without a record number | 0 | PASS |
exempt_class_leak |
Zero exempt-class rows | 0 | PASS |
dup_queue_keys |
No duplicate keys in the queue | 0 | PASS |
source_timestamps |
Every source timestamped | Both recorded | PASS |
violation_type_mix |
≥ 80% one dominant type (soft) | 100% | PASS |
active_count_delta |
≤ 40% swing against the prior pull | No prior pull | N/A |
Scroll the table sideways for the remaining columns.
The report above is the 2026-06 build, copied as it was recorded. It ships as a tab in the workbook and as a CSV, and the manifest records a SHA-256 for every file in the release.
§09 Refusals
Things this desk will not do
- No contact with building owners or occupants on a buyer's behalf. Any outreach is the buyer's own, under their own name and their own responsibility.
- No owner or occupant contact columns in an export by default. Where an export could carry a personal name, it is withheld pending review rather than shipped by default.
- Residential, sensitive, redacted, stale and utility-self-tested rows are suppressed from feed exports.
- One state's voluntary list of certified testers is off-limits under its own terms and is not scraped or used, even though it would be convenient.
- No claim of government affiliation, enforcement authority or non-public access, and no file that could be mistaken for an agency notice.
- No selling of a record class that current evidence does not support, however far along the build already is.
§10 Verify it yourself
Checking a row against the city, in four steps
This is the whole point of shipping identifiers with every row. For a New York City gas record:
- Open the city's public buildings portal.
- Search by the building identification number from the row.
- Open the property profile, then violations and notices of deficiency.
- Filter to gas piping and compare the record number, status and any penalty due.
During the July 2026 audit, one sampled row checked this way matched the dataset exactly: same record number, status active, civil penalty due $0.00. Do this before you contact anyone. Status moves daily.
§11 Corrections
When the evidence goes against the product
A record class gets re-described, re-scoped or withdrawn when checking says it should. Two cases, both from mid-2026:
The gas signal was re-described
A 20-record audit of the Queens queue on 16 July 2026 found all twenty sampled rows were active prior-cycle notices of deficiency carrying no penalty, with the next filing deadline at the end of 2026 — not confirmed current violations. The honest description is an active prior-cycle notice with no later filing publicly observed, and the wording used everywhere was changed to that.
A fire-protection premise was dropped
Records filed under a Manhattan sprinkler retrofit law looked like large installation work. Checking 81 of them against the city on 26 June 2026 found no confirmed installations: they were open paperwork on buildings already sprinklered. The premise was abandoned rather than sold as retrofit work.
If you find a row that is wrong, send the record number. Corrections are made to the build, not argued with.