L2 Market Data Under Bursts: Sequence Gaps, Recovery, and Fan-Out to Hundreds of Sessions

A handler that drops and reorders L2 updates under bursts is usually two separate failures behind one symptom. One lives on the ingest side, where the feed sequence is tracked and a gap has to be detected rather than absorbed. The other lives on the distribution side, where one normalised book is fanned out to many sessions and a single slow reader changes what the others receive.

They have different fixes, and applying the wrong one moves the symptom instead of removing it. What follows is the shape of the pipeline when the sequence has to survive a burst: what a gap actually is, how recovery joins a snapshot to a live stream, how fan-out decides ordering once, and where conflation is honest.

A sequence number promises order, not delivery

Feeds number their updates, and that number is the only ordering authority you have. Arrival time is not one: multicast paths reorder, several channels carry one instrument, receive queues are spread across cores, and a burst stretches all of it. A handler that orders by arrival is correct only while the network is calm - the condition nobody was worried about.

Six properties of the feed have to be known before recovery logic is written. Each changes what a gap means.

• The unit the sequence covers - channel, instrument, or book. A per-channel number does not tell you which instrument lost an update, and a per-instrument number does not tell you that a channel stopped
• The increment rule: strictly consecutive within the unit, or increasing with permitted holes. Both exist, and reading the second as the first produces recoveries that were never needed
• Whether numbers restart at a session boundary and what marks it - a restart read as a gap sends every instrument into recovery at the same moment
• Whether heartbeats carry the current sequence. Without them, a dead connection and a quiet instrument look identical
• Whether retransmission exists and over what window. If it does not, snapshot recovery is the only path back, and it has to be cheap enough to use often
• Which sequence number a snapshot is aligned with. Without it, a snapshot cannot be joined to a live stream at all

Where a property is genuinely unknown, measure it rather than encode a guess. Each of the six becomes a branch in the recovery path, and a wrong assumption there is found later as a book that quietly disagrees with the venue.

A gap and a reorder look identical for a few milliseconds

Both start the same way: the next update does not carry the number you expected. The difference is time, so the classification is not made on arrival but when a bounded wait expires.

• Out of order: you expected N, received N+2, and N+1 arrives while the wait is still open. Nothing is missing, and the only cost is the wait
• Duplicate or retransmission: a number at or below the last applied one. Dropped without touching the book, and counted, because a rising duplicate rate says something about the path
• Gap: the wait expired and N+1 never arrived. The book cannot advance past the hole, and this instrument goes to recovery
• Stale: the right number, too late to be useful. The bytes arrived, and downstream it is a loss

The wait is a bounded structure, not a queue that grows. It holds updates ahead of the expected number, keyed by sequence, so releasing them is a lookup rather than a sort.

• Release is a loop: apply the expected number, then apply what is already buffered while the numbers stay consecutive
• The deadline is expressed in time, not only in a count of pending updates - a burst fills a count-based window far sooner than the design intended
• Whatever the buffer waits, every consumer waits. Size the deadline from the reordering measured on your own path, not from a number that felt safe
• Buffer overflow is itself a gap declaration: the wait is bounded in memory as well as in time
• The wait is per instrument or per channel, never global. One quiet instrument must not hold back everything around it

Recovery is a snapshot joined to a stream you were already buffering

The join is the part that goes wrong. A snapshot is a book as of some sequence number, and it is stale the moment it is produced; what makes it usable is the incremental stream buffered while it was being fetched.

• Buffer the incremental stream before the snapshot is requested. A snapshot without a live stream behind it is already behind the market when it lands
• Read the sequence number the snapshot is consistent with. If the feed publishes none, the feed is snapshot-only in practice, and the design has to say so out loud
• Discard buffered updates at or below the snapshot sequence, then apply the rest in order. If the first of them is not the update immediately after the snapshot, the join failed and recovery restarts
• If the buffer fills before the snapshot arrives, restart the recovery instead of applying part of it - a partially applied recovery is indistinguishable from a healthy book
• Publish the instrument as degraded while it recovers, as an explicit state on the stream. A book with a hole in it, served as current, is worse than no book
• Verify after the join: the checksum the feed publishes, if it publishes one, or agreement between your folded book and the next snapshot

Recovery is a normal event, not an incident, and its cost belongs in the capacity plan: how long a snapshot takes to fetch, how much stream is buffered meanwhile, and how many instruments can recover at once before the snapshot service becomes the bottleneck.

Fan-out: normalise once, encode once, send many

Hundreds of terminal sessions want the same book. The mistake that multiplies under a burst is doing per-session work that is not per-session in nature: rebuilding a book for each subscriber, or serialising the same update once per socket.

• One writer per instrument shard owns the book. Readers never mutate it, which removes both the lock and the question of whose version is authoritative
• The writer publishes versioned updates into a ring buffer that readers follow at their own pace, so a reader falling behind slows nobody down
• Each update is encoded once per wire format and shared across sessions by reference. Only framing and flow control are per session
• Every session carries its own outbound sequence number, so a client can detect its own losses without knowing anything about the upstream feed
• Ordering is guaranteed per instrument, because that is the guarantee clients depend on. Ordering across instruments is either promised explicitly and implemented, or not promised at all
• Beyond one process, fan-out becomes a relay tier: each relay takes one subscription upstream and serves a share of sessions, so the work of the writer stays constant

Fan-out cost is decided by how many times an update is transformed, not by how many sockets receive it. Encoding once and passing a reference scales with sessions; rebuilding a book per session does not.

A slow consumer is a policy you choose, not an accident that happens

Somewhere there is a session on a bad network, or a terminal whose render loop stalled, and its outbound buffer fills. There are four possible behaviours, and two of them are chosen only by accident.

• Block the writer until the slow session drains: never. It turns one bad connection into a latency event for everyone on the shard
• Grow the queue without a limit: a slow consumer becomes memory exhaustion, and then an outage unrelated to the original session
• Bounded queue with conflation: correct for book state, where a client wants the current picture rather than every intermediate step
• Bounded queue with disconnect at a high watermark: correct for streams that cannot be conflated, where dropping an item drops meaning
• Whatever the policy, the queue is per session, and lag is measured continuously - queue depth, and the distance between the sequence published and the sequence written to the socket
• A disconnect states its reason. An unexplained close is retried in a loop; an explained one is followed by a resubscribe

Back pressure is where the two sides meet. If the outbound path can push back on the book writer, a slow terminal eventually delays the folding of the feed, and gap detection starts firing for reasons that have nothing to do with the venue. A bounded ring between the two stops that chain.

Conflation is honest for state and wrong for events

A book is state: the client wants the current levels, and a value already replaced carries no meaning of its own. A trade tape is a log of events, where each item is a fact that happened and cannot be summarised away.

• Conflate: price level updates, top of book, aggregated depth, and derived statistics such as last price or session volume
• Do not conflate: trades and prints, order and execution reports, auction and phase changes, and anything a client aggregates over time - a tape built from a conflated stream is a wrong number held confidently
• Conflate per key, not per stream. Keeping the latest update for each price level preserves the book; keeping the latest update overall throws away every level that did not change last
• A conflated update carries the sequence number of the state it represents, so a client knows the point it corresponds to
• The conflation interval is part of the latency you report. A stream conflated on an interval is not described by the latency measured on the unconflated one
• A client that needs every intermediate state - a backtest, a compliance record - takes the unconflated stream and pays in bandwidth

Reconnect is a resynchronisation, and they all arrive at once

When a session comes back, the book it holds is worthless unless the server can prove continuity. The default is a fresh snapshot per subscription, with its sequence number, applied to a client that dropped its local state first.

• Resume from a sequence number is offered only where a bounded replay buffer exists. When the requested number has aged out, the server says so and falls back to a snapshot rather than sending a stream with a hole
• Session state across a reconnect is an explicit decision: either the server keeps subscriptions for a bounded time under a session token, or the client restates them on connect. Both work; an implicit mixture does not
• Duplicate delivery after a resume is expected, and the client discards by sequence. At-least-once plus sequence numbering is easier to implement correctly than exactly-once
• Reconnects arrive together, because whatever disconnected one session usually disconnected many. Jittered backoff on the client and admission control on the server keep the recovery from becoming the second outage
• Snapshots for that crowd come from a per-instrument cache refreshed on a cadence, so the writer serialises a snapshot on a schedule rather than once per reconnecting session
• The client-side book is rebuilt, never patched. A terminal that keeps its old levels and applies new deltas on top carries the pre-disconnect error into a book that now looks fresh

The failure worth designing for is not a single reconnect. It is a network event that returns hundreds of sessions in the same second, each asking for a snapshot of every instrument it was watching, while the ingest side recovers from the gap that same event produced.

What amBrain can substantiate publicly: we build low latency trading platforms, matching engines and real-time bidding systems in Rust from Yerevan, Armenia, and the market data latency we publish - under 5 ms - is measured on the paths we build. If your handler is losing sequence under bursts, the conversation worth having is the one that separates the ingest side from the distribution side before either is rewritten.