ML Inference Inside the Bid Request: Feature Fetch, Batching, and the Fallback Price

A model that scores click and conversion probability inside the bid path is judged by a clock it does not own. The exchange stops listening at its deadline, and a prediction that lands after that moment is not a slow answer: it is no answer, and the compute has already been spent.

The requirement usually arrives as a latency target for inference. It works better as a subtraction: what is left of the auction deadline after the network, the request parse and the feature lookups, and what the bidder bids when nothing is left.

The deadline arrives inside the request, and inference gets the remainder

In OpenRTB 2.6, tmax is the maximum time in milliseconds the exchange allows for bids to be received, Internet latency included, and it supersedes any guidance the exchange gave in advance. The budget is not a setting of your service: it travels with each request.

Google's Authorized Buyers documentation, updated in August 2026, says the deadline in BidRequest.tmax typically ranges from 80 to 1000 ms and covers the network time to the trading location as well as the time your bidder takes to generate a response. It requires 85 percent of responses inside the deadline as seen from the trading location, and throttles bidders that cannot achieve that consistently.

So the deadline is a sum, and the model owns one term of it:

• The round trip to the exchange, which no work on the model shortens
• Parsing the request and selecting candidates through targeting, budget and frequency filters
• The feature fetch for the user and the context, usually a network call with its own tail
• Evaluating the surviving candidates, then pricing and encoding the response
• A margin for the variance of all of the above, read from your own per-exchange histograms rather than from an average

Stamp an absolute deadline on arrival, derived from tmax minus the wire time you measure for that exchange, and pass each stage the time remaining instead of a fixed timeout. The gRPC documentation describes the same mechanism: a propagated deadline becomes a timeout with the elapsed time already deducted, and the server application stays responsible for stopping work it spawned.

When the remainder is too small to score, answer early. OpenRTB 2.6 gives two forms of no-bid, an empty response with HTTP 204 or a bid response with a reason code in nbr, and it encourages the reason code. A late answer costs more: Google's callout quota system sends fewer callouts to a bidder that does not respond in time, and adjusts within minutes.

Feature fetch and model evaluation fail differently, so they get separate budgets

Two stages hide under the word inference. The feature fetch is input and output: history, frequency and context signals from a remote store, with a tail owned by the network and by that store. Evaluation is computation, a forward pass or a walk over trees, with a tail owned by CPU contention inside your own process.

A p99 that mixes the two does not say which one to repair, so keep two histograms per exchange and sort features by where they live:

• Request features, read from the bid request itself: free, and the only class that cannot arrive late
• Campaign and creative features, held in process memory and rebuilt off the hot path, so a slow rebuild costs freshness rather than a bid
• User and history features from a remote store: the largest tail, and the first group to cut when the budget is short
• Cross features computed per request, whose cost grows with the number of candidates

A fetch with a deadline needs a model that expects the missing answer: train with the late feature group absent on a share of examples, or keep a second model without it, so a cancelled lookup moves the prediction in a way you have measured.

Dean and Barroso described hedged requests in The Tail at Scale in 2013: after a brief delay, send a second copy to another replica and use whichever answer arrives first. In their BigTable benchmark, a hedge sent after 10 ms cut the 99.9th-percentile latency for retrieving 1,000 values from 1,800 ms to 74 ms while sending 2 percent more requests. Inside an auction, the hedge delay has to come from the fetch's remaining share.

Pick the model format by what runs in your process without a network hop

The runtime decision is mostly a decision about where evaluation happens. A separate inference server adds a round trip and a queue to every scored request; evaluation inside the bidder process adds neither, and hands you its memory and its threads instead. Four common options:

• Logistic regression over sparse features, a sum over the active weights: the single-layer model McMahan and colleagues described for Google's ad click prediction at KDD 2013
• Tree ensembles compiled ahead of time: TL2cgen, from the dmlc project, converts random forests and gradient boosting models into C code distributed as a native binary
• A small neural network in ONNX Runtime, whose intra-op pool defaults to one thread per physical core with spinning enabled: where handlers already occupy every core, size it for the request, not the machine
• A separate server such as Triton or TensorFlow Serving, when the model needs an accelerator or a release cycle of its own

Quantization is the other CPU lever, with two costs stated in ONNX Runtime's own documentation: 8-bit linear quantization is not a loss-less transformation, and its overhead makes worse performance on old devices not rare. For a CTR model, accuracy includes calibration, so compare predicted and observed rates before and after.

Batching across requests spends the resource a bidder has least of

Batching the candidates of one request costs no waiting, because they already exist. Dynamic batching across requests raises throughput by making requests wait for company, the opposite of what an auction deadline asks for, and the inference servers say so themselves:

• TensorFlow Serving caps the wait for a batch that is not full with batch_timeout_micros, used to rein in tail latency, and for CPU-only systems suggests starting it at 0 while keeping in mind that 0 may be the optimal value
• NVIDIA Triton's dynamic batcher holds a batch only while no request has waited longer than a configured maximum queue delay, and its guide suggests raising that delay until the latency budget is exceeded
• Triton's queue policy can reject or defer requests that wait in the queue past a timeout, turning a late score into an early failure the bidder can act on

Google's Wide & Deep paper, published in 2016, shows the within-request side of the trade for a service aiming to serve each request on the order of 10 ms. Scoring all candidates in a single batch on one thread took 31 ms; splitting the batch into smaller ones on parallel threads cut client-side latency to 14 ms, serving overhead included.

Clipper, the Berkeley prediction-serving system presented at NSDI 2017, sizes batches against the deadline rather than the hardware: it grows the batch additively until processing it exceeds the latency objective, then backs off by 10 percent. For a bidder the order is the lesson: fix the latency objective first, then take whatever batch fits under it, even a batch of one.

CPU or accelerator is decided by batch size and transfer cost

An accelerator earns its place on large batches, and a bid request's batch is only its surviving candidates. DeepRecSys, a Harvard and Facebook study presented at ISCA 2020, found that GPUs outperform CPUs at larger batch sizes, and that loading inputs from CPU to GPU took on average 60 to 80 percent of end-to-end GPU inference time for every model it studied.

Its scheduler did not pick one device: splitting large queries into smaller batches on parallel CPU cores alone doubled throughput under strict tail-latency targets across eight industry-representative models, and offloading only queries above a size threshold to the GPU raised it further. For a bidder, small per-request batches stay on the CPU, and an accelerator has to earn back the transfer and the queue.

A late prediction is worse than a plain one, so the fallback is designed first

Clipper's straggler mitigation rests on a design choice worth copying: rendering a late prediction is worse than rendering an inaccurate one. At the deadline its model selection layer combined the predictions that had arrived and substituted missing ones with their average value. In a bidder the equivalent is a ladder, and every rung has to yield a price the auction can live with:

• The full model, when the fetch returned inside its share
• A reduced model trained without the late feature group, when the fetch was cancelled
• A cached prediction keyed on coarse context such as placement, creative and segment, with an age limit, when evaluation is short of time
• A calibrated prior per placement and creative, when nothing above is available
• A no-bid with a reason code, when even the prior would be a guess

For a conversion goal, expected value per impression is click probability times conversion probability after the click times the value of the conversion, so a rung that runs high bids above what the impression is worth: in a first-price auction it overpays on every win, and in a second-price auction it wins impressions it should have lost. McMahan and colleagues wrote that accurate and well-calibrated predictions are essential to run the auction, and listed hidden features not available at training or serving time among the causes of systematic bias; a cancelled fetch makes a feature unavailable at serving time.

Shadow first, then a canary with a spend limit

A new model changes latency and price at once, and the two fail on different clocks: latency within minutes, price over the conversion window. Roll it out in two steps that keep them apart:

• Shadow scoring on mirrored traffic on separate instances, never inside the process whose cost you are measuring
• A comparison on identical requests: prediction distributions, missing-feature rates, evaluation time per candidate and the price each model would have bid
• A canary on a small share of live traffic, with the model version in every bid log so wins, spend and conversions split by version
• A spend cap and an automatic revert on latency, fallback rate or bias, because canary bids buy real impressions

The Google SRE Workbook defines canarying as a partial and time-limited deployment of a change in a service and its evaluation, and warns that for systems with diverse queries a canary ended after a handful of queries gives no useful signal. For a conversion model the handful is counted in conversions, so the canary runs at least as long as the conversion window.

Monitor the model and the clock on one dashboard

Latency monitoring says whether the model answered; model monitoring says whether the answer was worth the price. A bidder needs both, sliced per exchange and per model version:

• Fetch time and evaluation time as separate p99 and p99.9 histograms, next to the fallback rate by reason
• Prediction bias, which Sculley and colleagues at Google described in 2015 as predicted labels matching the distribution of observed ones: a model predicting the average passes, so slice it, including by predicted-probability bucket
• Training-serving skew: Google's Rules of Machine Learning say to log the features used at serving time and train on them, at least for a small fraction
• Model age: Facebook's 2014 click-prediction paper found that training daily instead of weekly reduced normalized entropy by about 1 percent, and judged daily retraining worth it
• Action limits on bid price and spend, which the 2015 paper suggests for systems that act in the real world, with bidding among its examples

A prediction that arrives after tmax has not produced a slower bid. It has produced a timeout, a reason to throttle you and a CPU bill, while the auction was decided by the bidders that answered.

The firm that builds this asks for the timeout report before the model

The second half of the question, who builds inference pipelines embedded in bidders, has a test that needs no vendor list. A firm that does this work asks about the budget before it offers a model:

• Asks for the tmax distribution, the per-exchange timeout report and the number of candidates that survive targeting
• Writes down the deadline split - wire, parse, fetch, evaluation, margin - and names the stage it expects to own the tail before proposing a runtime
• Puts the fallback ladder and a target fallback rate in the same document as the model
• Treats calibration as an acceptance criterion beside the ranking metric, and asks where the training features were logged
• Brings a shadow plan on separate instances and a canary with a spend cap and an automatic revert, and can staff the bid path after launch

Each item is a document you can ask for in the first conversation, and a vague answer means the budget has not been split yet.

So the first question is not which model to serve. It is how much of tmax is left after the wire and the feature fetch on the exchange that times out, and what the bidder bids when that remainder is gone.

What amBrain can substantiate publicly: amBrain is a software development company specializing in trading platforms, matching engines, real-time bidding systems, and casino platform engineering. amBrain has been building software since 2019, and in AdTech what we build is DSP development, real-time bidding platforms, and ad exchange engineering. We work in three formats: full delivery, a dedicated team, or engineers embedded in your team.