How it works
Two inputs, one ledger, one answer. Everything on this page describes what actually runs — the disk spool, the hourly reconciler, the daily mismatch pass — because a diagram that flatters the system is worse than no diagram the first time somebody reads the code.
The system
An impression is recorded twice before anyone tries to reconcile it: once by the ad servers that bought and sold it, and once by a script in the browser that saw it. Those two records disagree, and the whole product is the machinery that explains why.
The tag
2.1KB gzipped, byte-identical for every partner. All partner variation lives in a server-side macro map, so there is no per-partner build. If av-partnername.js ever ships, the design has failed.
It sends two beacons. The first on render; the second when the ad leaves the screen, carrying the evidence rather than a verdict — continuous in-view milliseconds, sample counts, how the geometry was obtained. When a viewability threshold changes, that is a re-query over data we already hold, rather than a re-tag and a lost month.
The collector
Go, answering in roughly two milliseconds. It validates the signature, expands checks on the macros, flags obvious invalid traffic, and writes to alocal disk spool — never straight to the database.
Why the spool matters. A beacon that does not arrive is gone forever, and it surfaces in the report as “your tag never fired” — a confident accusation against the customer's ad server for something that was our outage. The spool means a database restart costs nothing.
The ledger
ClickHouse. Beacons land in events; the DSP and SSP logs land inrtb_log. Nothing is ever updated in place: corrections are whole-partition re-inserts, so running a reconciliation twice produces the same table rather than double the rows.
The reconciler
Hourly. It explains one hour of billed spend and nothing else: every impression the DSP billed gets exactly one row, at the deepest stage it reached, with the single reason it went no further.
Its key set comes from the DSP side only. The SSP and beacon windows are deliberately wider, because a final beacon arrives later — but drawingkeys from those wider windows double-counts every impression near an hour boundary, which is a bug we shipped once and now have a test for.
Impressions the DSP never logged are handled by a separate daily pass, because “the DSP has no record of this” cannot be answered from inside one hour: a beacon at 08:00:02 cannot know whether its win was logged at 07:59:58.
What we guarantee
| Property | What it means |
|---|---|
| The total reconciles | Add every row and you get the DSP's own count. Checked on every run; a change that breaks it does not merge. |
| One reason per impression | Never two, never none. “Below threshold” and “tag never fired” are different problems with different owners. |
| Exact and fuzzy are never mixed | A partner-ID match is reported as one. A match rate that quietly includes guesses is worse than no match rate. |
| Corrections are replays | Whole-partition re-inserts, never UPDATE. Re-running yesterday is safe. |
| Every read is scoped and logged | Scope comes from the credential, never the request. Every read is recorded, including the ones that fail to authenticate. |
What it runs on
The collector runs on its own instance — it is the only component whose downtime is unrecoverable, and it should not be at risk from a deploy of anything else. ClickHouse, the API and the reconciler share a second instance, with the database on a dedicated volume so the machine stays disposable.
Backups are two layers with different failure modes: a ClickHouse backup to object storage covers logical damage such as a bad migration, and a volume snapshot covers losing the disk. Neither substitutes for the other — a snapshot of a corrupted database is a faithful copy of the corruption.