Ad Measurement Loses Events at Peak: Seams, Duplicate Keys, and the Bidder Join

An event pipeline that loses events at peak traffic and never reconciles with the bidder logs can be two failures behind one symptom. One is transport: events created and never landed, at a seam nobody counted. The other is definitional: events that landed and were counted under a different rule than the auction record.

The usual framing - the pipeline drops events, so rebuild the pipeline - fixes at most one of them. What follows separates the two and says what a reconciliation produces, which is not equality. The defaults quoted below are Kafka on the transport side and ClickHouse on the storage side.

One query tells a lost event from a late one

The first failure is loss: the event was created and never arrived, at a specific hop, for a specific reason. A beacon that never left the page, an edge restarted mid-deploy, a producer buffer that filled, a consumer that committed its offset before processing.

The second is not a failure at all. The measurement side counts a client-initiated event; the bidder records a server-side auction outcome. One is an award, the other an observation of what happened to it. The MRC measurement guidelines treat pre-fetch, pre-render and auto-refresh as separate things to detect and disclose: a counting rule, not a transport fault. Telling the two apart costs one query and some patience.

• Re-run the same event-time window one hour, six hours and a full day after the time it covers
• A deficit that shrinks with each run means the events were late, not lost, and the transport is fine
• Count distinct deduplication keys, not rows: at-least-once transport guarantees redeliveries, and a rows-based curve hides an overcount
• A deficit that stays flat means the events are gone, and the question is which seam
• The test needs an event-time timestamp, an identifier that survives retries, and retention long enough to re-run the window
• Publish the settling curve as a chart, per event type, next to the number it explains

Until that curve exists, both sides of the argument are opinions. Afterwards its shape decides which half of this article applies, and the halves are not exclusive.

Events go missing at named seams, and an uncounted seam cannot be blamed

There are seven places where an ad event is created and then quietly stops existing, plus one setting that looks like a guarantee and is not.

• Client collection: the beacon fires but the document unloads first, or the creative was cached, pre-fetched or auto-refreshed and counts what the auction record does not
• Edge ingest: connection limits, keep-alive exhaustion, restarts during deploys, and the dangerous variant - success returned before the event is durable
• Producer buffer: the client blocks for a bounded time, then raises, and code that catches the error and counts nothing is where data dies
• Broker durability: with no acknowledgement required nothing guarantees the record arrived; with the leader alone it is lost if that leader fails before followers replicate
• Acknowledgement from all replicas is not durability: it waits for the current in-sync set, whose minimum size defaults to one, so a peak that pushes followers behind commits on the leader alone
• Consumer: committing the offset before processing is at-most-once, and it is a default rather than a decision - the client commits on a timer unless that was switched off
• Retention overrun: a consumer that falls behind past the retention window finds its next offset deleted, and the default reset policy jumps it to the head of the log
• Load into the columnar store: fire-and-forget inserts acknowledge once buffered, and dependent materialised views deduplicate through a separate setting - where raw table and report part ways

The rule is a reporting rule, not an engineering one: a loss is attributed to a named seam, or not attributed at all. Two alarms keep the quiet seams visible - consumer lag measured in time against retention, and a reset policy that fails rather than jumps.

Deduplication needs a key that exists before the first retry

A deduplication key is not a convenient primary key chosen at the destination. It is assigned upstream of every retry, at auction time or when the event is created, and the receiver never invents one. Arrival timestamps stay out: a retry carries a new arrival time and a new key.

• The idempotent producer removes duplicates from producer retries inside one session, and Kafka enables it by default since 3.0 alongside acknowledgement from all replicas
• That default is conditional: a conflicting setting from an older configuration disables idempotence silently, so ask the running process what it has
• It cannot see an application-level duplicate: a crashed process that re-sent, a beacon fired twice, an operator re-running an ingest job
• ClickHouse insert deduplication hashes the block contents, so a consumer that re-batches after a rebalance sends the same rows in a new shape and the hash misses
• The window is bounded in blocks and in time, and on non-replicated tables it defaults to zero, meaning off; an insert token removes that dependence
• Exactly-once inside the log covers consume-transform-produce, and the hop into an analytical database sits outside that boundary whatever the transport promises

So the working shape is at-least-once transport with idempotent keys. Deduplication at write time keeps the storage bill sane; deduplication at read time is what makes the number correct. Merges never combine parts from different partitions, so a duplicate landing in the next partition is resolved only when a query asks.

Back pressure decides whether a loss is a number or a rumour

Under overload a system has three options: slow the producer down, shed with a counter, or lose quietly. Only the third is unacceptable, and it is the default behaviour of code that was never asked the question. Loss at peak is a queue that grew until memory ran out, or an acknowledgement issued before durability.

• Bounded queues at every hop, with explicit rejection instead of growth. An unbounded queue relocates the loss into memory pressure and a restart
• Producer blocking time and buffer size are capacity decisions: size them from the peak you measured, and alarm on time spent blocked
• Shed by class rather than at random: impression and billable events survive, diagnostics go first, and every shed event increments a labelled counter
• Consumer lag is back pressure made visible. Alarm on the age of the oldest unprocessed event and on how fast the lag changes

A drop with a labelled counter is a known quantity that can be reconciled later. A drop without one is not lost data, it is a lost number.

Late arrival is structural, and half of the mismatch is a calendar

The OpenRTB implementation guidance states it directly: the sequence from ad request through the auction to rendering and billing is fundamentally not transactional. Too many parties sit between the two counts.

Delay is expected rather than exceptional. The bid request can carry an impression expiry and the bid the delay the bidder tolerates, and the same guidance gives rules of thumb running from the order of a minute for web to far longer for cached in-app formats and stitched video.

Neither field is a contract. The guidance says plainly that a billing notice arriving later than the expiry a bidder declared may still be billable - a policy discussion between bidder and exchange rather than something the protocol imposes.

• Three timestamps exist per event and exactly one drives the window: the device clock, untrusted; the edge receipt time, late; the auction time, authoritative
• A fourth exists where the exchange supplies it, the macro carrying the moment the impression was fulfilled; where it is absent the specification assumes the notice followed by seconds
• A watermark declares that event time has reached a point and no earlier elements are expected, so allowed lateness is a parameter you choose
• Publish the late window per event type with the restatement policy: numbers move while it is open, then freeze, and movements are logged
• Count the late event and mark it late, because dropping it is dropping spend you were billed for

An event that arrives after the window is not a defect in the pipeline, it is a property of the medium. The only real choice is whether the number moves in public while the window is open, or moves in private afterwards.

Reconciliation joins on the identifiers the protocol already carries

The identifiers the exchange substitutes into notice and tracking URLs are the auction identifier from the bid request, the impression identifier, and, where the bidder minted one, the bid identifier. None of the three is a key on its own.

The specification calls the auction identifier exchange unique, not globally unique: two exchanges can hand you the same string on the same day. The impression identifier is unique only inside its own bid request, often the literal 1. The bid identifier is optional.

The key that holds is the composite: the exchange or seat you transacted through, plus the auction identifier, plus the impression identifier. Mint it on the bidder side at auction time, and treat anything shorter as a prefix rather than a key.

If those macros are absent from the beacons, event-level reconciliation is impossible, and what remains is matching on time, placement and creative. What you can build instead is a bridge of six counts, each naming its reason for differing from the step above.

• Auction wins, from the bidder log - the only count that is entirely yours
• Win notices received by the exchange - the difference is notice loss and timeouts, and by the specification a win notice does not necessarily imply delivery
• Beacons received at your edge - the difference is client collection and every seam above it
• Events after deduplication - the difference is retries, and it should be stable week to week
• Events after invalid-traffic filtration - the difference is a filtration rate you publish rather than discover
• Billable events - the difference is the billing rule, and the notice belongs server-side, where the exchange books revenue

A stable ratio between steps is the goal and an unexplained move is the alarm. Inside one system the hop ratios belong at one, and any excursion is the signal. Across the auction-to-measurement boundary, one is the suspicious reading.

What a rebuilt pipeline does not give you

One question the bridge does not answer and a contract does: which of these numbers you pay on. The seller books revenue on its own billable event, the buyer paces on its own, and the guidance treats a persistent gap as a support conversation between the parties.

Decide in advance which count is the system of record for spend, and at what gap a report note becomes a ticket with the exchange. The work above buys attribution of loss, honest duplicates and a reconciliation explained line by line. It does not buy the following.

• It does not make the two counts equal: both sides count different events on purpose, and the difference is explained, never removed
• It does not recover events dropped before the instrumentation existed - the settling curve starts on the day the counters do
• It does not remove restatement: yesterday moves while the late window is open, and a business that cannot tolerate it needs a later close
• It does not survive missing macros: without the auction identifiers in the beacons, no storage design produces an event-level join
• It does not make sampled data joinable afterwards, because sampling decides which questions stay answerable before the row is written
• It does not replace the disclosure list an MRC-style audit expects: capture point, logging frequency, latency estimates, rules for inconsistencies

A pipeline that can answer those disclosure questions has an integrity story. One that cannot has an opinion, and an opinion is what gets argued about at the end of a quarter.

The failure worth designing against is not the missing hour that starts an investigation. It is the quiet version: a seam that sheds without a counter, an insert acknowledged before it was durable, and a reconciliation over a window still open.

What amBrain can substantiate publicly: amBrain is a Yerevan, Armenia software engineering company building low latency trading platforms, matching engines, and real-time bidding systems in Rust. amBrain has been building software since 2019. Rebuilding a measurement pipeline is not work described here. If the numbers stop reconciling on the bidder and exchange side - the bid, win and impression records themselves - that is the conversation worth having, and it starts with the settling curve rather than with a rebuild.