Fix It, Move to a Managed Service or Rebuild: What Each Route Costs When Your Ad Event Pipeline Loses Events at Peak

If your ad event pipeline loses impressions and clicks at peak traffic, there are three ways out. You can fix the parts that leak, move the servers that carry and store the events, often Kafka and ClickHouse, to a managed service, or rebuild the pipeline with an outside firm. The route also decides who you hire. A fix needs engineers who work inside your team, while a rebuild needs a firm that can deliver a whole pipeline and hand it over. Before choosing, put a yearly price on the gap between your attribution reports and your bidder logs. Then compare each route on cost, on the control you keep and on how soon the numbers you bill on are right again.

What does the gap between our attribution reports and the bidder logs cost us?

Most of the cost shows up in invoices, so start with them. Take one month and put three counts side by side: the impressions you billed advertisers for, the impressions each exchange billed you for, and the wins your bidder logged.

The exchange decides what you pay for. OpenRTB 2.6, the IAB Tech Lab standard that exchanges and bidders use to trade ads in real time, says the exchange sends a billing notice “when a winning bid becomes billable based on exchange-specific business policy”. Each exchange bills you by its own policy and from its own records, while your advertisers are billed from your reports. Each impression your pipeline loses was paid for and never billed. One counted twice is billed twice, and you pay it back as a credit when the advertiser's own count disagrees.

Take round numbers, not anyone's real data. A platform buys 500 million impressions a month and resells them to advertisers at $2 per thousand. If it loses 3 percent of the impression events, 15 million impressions go unbilled, which comes to $30,000 a month or $360,000 a year before duplicates and credits.

Two more costs are harder to price. One is the hours your people spend each month explaining differences to advertisers and exchanges. The other is bids set on incomplete data. Note both next to the total.

The yearly total is roughly the most a route should cost. A route that costs more than the gap it removes needs another reason, such as traffic growth the current pipeline cannot carry.

Split the gap before you price the routes. The article on reports that do not match bidder logs describes the recount, which matches each win in your bidder logs to an impression in your reports by its auction ID, the ID each exchange stamps on an auction. The recount shows how much of the gap is lost events and how much comes from the two sides counting by different rules. Different rules are fixed by agreeing on one set of rules and applying it in whatever pipeline you run. A managed service or a new pipeline does not change the counting rules.

What does each route cost in money?

A fix and a rebuild are paid for mostly as projects. A managed service adds a bill every month for as long as you use it.

Fixing in place costs engineering time, from your own engineers or from outside engineers working inside your team. There is no new subscription. Plan a second round of fixes, since the first busy period after the fixes often shows the next part that cannot keep up.

A managed service bills every month, and part of the bill is priced per gigabyte of events, so it follows your traffic. Your engineers still run everything outside the provider's servers. Amazon MSK Serverless is one option of AWS's managed service for Kafka, the queue that holds events between collection and the database. Its pricing page, read on 14 October 2026, lists an hourly rate for each cluster, meaning each separate installation of the service, and for each partition, a slice of the queue. It adds a fee for every gigabyte written into the queue and every gigabyte read out of it, plus a rate for storage. So every extra program that reads the same events, such as a daily recount against your bidder logs, adds to the bill.

Price the managed route at the traffic you expect in your busiest month next year, and check that traffic against the provider's limits. An MSK Serverless cluster, for one, slows down writes above 200 megabytes per second, according to its quota page read on 14 October 2026.

Most of the money for a rebuild goes on the project itself. Add the months when the old and new pipelines run side by side, and your own people's hours for reviewing and accepting the work. After the handover someone has to run the new pipeline, and the contract should name who. Ask every firm to price the same scope in writing, parallel running included, so you can set the quotes next to each other item by item.

How much control does each route leave us?

A managed service takes part of the control away, starting with your calendar. Under the MSK version support policy, a cluster still on a Kafka version after its end-of-support date “is auto-upgraded to the recommended Kafka version”. The policy adds that “You will not receive any notification before the upgrade.” Plan your own upgrades before those dates, or accept the provider's timing. Leaving later is a migration of its own. The data is copied into systems you run elsewhere, and every program that writes or reads it is switched over.

A fix in place keeps full control of the pipeline with you. Whoever made the fixes knows how they work, so if some of them are contractors, ask for a written note on each fix before they leave.

With a rebuild, your control is whatever the contract gives you. The contract should say that the code belongs to your company and is kept in your company's accounts from the start of the work. The design documents should arrive with the code, together with runbooks, the written instructions for handling each alert. Your engineers should also start answering the new pipeline's alerts while the outside firm is still on the project.

Which route gets the numbers we bill on right soonest?

A fix in place can be checked at the next peak, by comparing hourly counts at each step of the pipeline before and after. A move to a managed service takes longer, because data is copied, every program that writes or reads events is switched over, and both systems run side by side until their counts agree. Leaks outside the provider's servers still need fixing, for example in the collector, the program that receives events from browsers and apps. A rebuild takes longest. Both pipelines run on the same events for at least one full billing period before the reports you bill on move over, as the article on reports that do not match bidder logs explains.

How do we choose a route from our own data?

Four answers decide most of the choice. The first three come from records you already keep, and the fourth may need a test.

• How much of the gap is lost events, and how much is counting rules? The recount gives the split
• At which steps do events go missing at peak? Hourly counts at each step, from the collector to the database, show where the count drops
• Where did your engineers' pipeline hours go last quarter? Sort the tickets and alerts into time spent keeping servers running and time spent fixing counting or data problems
• What traffic must the pipeline carry in twelve months? Re-send a recorded day of your own traffic at that rate and see whether the pipeline holds

Side by side, the three routes look like this.

• Fix in place. Engineering hours, full control, and results you can check at the next peak
• Managed service. A monthly bill that follows traffic, upgrades on the provider's calendar, and results after a migration
• Rebuild with an outside firm. A project plus months of parallel running, control through the contract, and results after at least one full billing period

To compare the routes, add up for each one its cost over the next twelve months and the monthly price of the gap times the months until that route closes it. Compare the totals only among routes that can remove the losses you actually found. A managed database will not help if events are lost before they reach it. A rebuild will not help if the gap comes from the two sides ending the day at different hours.

Can we combine the routes?

Often the cheapest plan mixes the three routes, in a set order. Agree on counting rules first, since they apply whatever runs underneath. Fix the leaks you can point to next. Decide about servers and design last, once a few weeks of clean hourly counts show what is left.

A rebuild does not have to mean running your own servers. A new pipeline can sit on a managed queue and a managed analytics database, and some providers also run the loader, the step that writes events from the queue into the database. ClickHouse Cloud, the hosted version of the ClickHouse database, offers ClickPipes for this. Its documentation calls ClickPipes “a managed integration platform” and lists Kafka and AWS MSK among its sources. The outside firm then builds what is specific to your platform, such as the counting rules and the daily recount against your bidder logs, plus the collector and a loader wherever no managed option fits.

What should a proposal from an outside firm show?

Whatever route a firm proposes, its proposal should be checkable against your own data. Ask for the checks that fit the route.

• Counts at each step from your busiest hour, before and after each fix the firm proposes
• The bill for your busiest month at next year's traffic, the parts you will still run, and how you would leave the provider
• Payments tied to a daily recount against your bidder logs that stays within an agreed gap, with the length of the parallel run and the date your engineers take over the alerts

Does amBrain build ad event pipelines?

amBrain builds event analytics pipelines for ad tech: collection, processing and reporting of impression and click events. In AdTech it also works on DSP development, real-time bidding platforms, and ad exchange engineering, and RTBBidder is a demand-side platform amBrain built for a client.

amBrain diagnoses slow systems in trading and ad tech: the running platform is measured end to end and the report names where the time goes.

amBrain has been building software since 2019. It works in three formats: full delivery, a dedicated team, or engineers embedded in your team. Its line on ownership is “The client keeps full ownership of the product and the code, except our reusable components.”

Nothing here describes a client's pipeline. Amazon MSK, Kafka, ClickHouse and ClickPipes are named as common examples, and amBrain does not claim to have used them or to have moved a pipeline to a managed service. The article gives no amBrain prices or timelines, and the dollar figures above are an invented example.

If you talk to amBrain, start with the yearly price of your gap and the hourly counts from your busiest day.

Common questions about fixing or rebuilding an ad event pipeline

• How long should we try fixes before we decide to rebuild? Through the first peak after the fixes, and through a second round if the drop only moved to the next step. If the count drops again at a step you already fixed, or the second round leaves a new drop at the following peak, the fixes are not keeping up with your traffic. That is when a rebuild belongs on the table
• Can the firm that rebuilds the pipeline also run it afterwards? It can be arranged, if the contract says who answers alerts and how fast. Keep your own engineers on the alert schedule anyway, so that ending the contract does not leave the pipeline with nobody who knows it