Slow Order Execution at a Small Prop Trading Firm: Where the Time Goes and Who Can Fix It

Slow execution gets fixed by whoever owns the part of the order path where the time is lost, so the first job is to find that part. Timestamp each order at the points you can see, and the largest gap tells you who to call: your developers or an engineering company for delays inside your software, a hosting provider for distance, the broker for delays on its side.

What does “our order execution is too slow” mean?

The complaint can mean four problems, and each has a different owner.

• Every order is slow. The delay is about the same on a quiet morning and at the open. That points to a cost every order pays, such as distance, the way you connect to the broker or slow work your own software does before each order leaves
• Orders are fast until the market gets busy. At the open or on news, the delay jumps. Orders are waiting in a queue somewhere, behind a program that cannot keep up, a message limit or a machine busy with other work
• Orders arrive on time but fill late. A limit order rests in the book until someone trades against it, and in a fast market the price moves away. Timestamps show whether the order was late. They cannot show what price was available
• The screen is late. If prices on the trading screen lag, traders click late and blame execution. That is a market data problem, covered in a separate article on this blog

Only timestamps on real orders, and on the prices they reacted to, tell these four apart. Use traders' complaints to choose which days to check.

Where do the milliseconds go between our system and the exchange?

An order crosses four stretches on the way out, and the acknowledgment comes back from the broker, sometimes only after the exchange has accepted the order.

• Your side. The strategy or trader decides, the order is built, your own checks run and the order is sent. Delays come from work done before sending, program pauses, network settings or a busy machine. Your developers or an engineering company can change this part
• The wire. The order travels from your server to the broker's entry point. Distance, internet routing and extra hops such as a VPN add the delay here. A hosting or colocation provider, or a network engineer, can shorten it
• The broker. Its gateway takes the order, runs its checks and routes it to the exchange. Delays come from the broker's own systems and its required checks. Only the broker can change them; you choose how you connect and which broker you use
• The exchange. The matching engine accepts the order and sends back the acknowledgment. Little time goes here: Nasdaq states a round trip from order to acknowledgment of under 50 microseconds on its high-speed 10G colocation network. Nobody you can hire changes this part; you can only get closer to it

In the US, the broker's checks are not optional. SEC Rule 15c3-5 requires a broker with market access to “prevent the entry of orders that exceed appropriate pre-set credit or capital thresholds” and to reject orders “that exceed appropriate price or size parameters”. The same rule puts those controls “under the direct and exclusive control of the broker or dealer”. You can ask a broker how long its checks take, but the rule does not let it switch them off.

How do we find out where the time is lost?

Record four timestamps for every order:

• Decided: the strategy or the trader chose to send it
• Sent: the order left your server
• Acknowledged: the broker's confirmation that it accepted the order reached your server
• Filled: the fill arrived

Decided to sent is your software. Sent to acknowledged is the wire and the broker, there and back, plus the exchange if the broker waits for it. Ask the broker which way it works. For an order resting in the book, the time from acknowledged to filled is mostly the market.

Measure one more number: the network round trip from your server to the broker's entry point. It tells you how much of sent to acknowledged is the wire.

FIX is a message standard for trading, maintained by the FIX Trading Community. If you connect over it, the broker's messages carry two timestamps, one for when the message was sent and one for when the event it reports happened. The FIX specification calls these fields SendingTime and TransactTime, and the broker can tell you whose clock sets each. Next to your own timestamps, they show which part of the round trip happened on whose side.

Comparing your timestamps with the broker's works only if both clocks are right. FINRA Rule 6820 requires broker-dealers that report to the Consolidated Audit Trail to keep business clocks within 50 milliseconds of the NIST atomic clock. Under EU rules, a member of a trading venue that uses high-frequency algorithmic trading must keep its clocks within 100 microseconds of UTC. A clock allowed to be 50 milliseconds off cannot locate a delay of a few milliseconds.

Sent and acknowledged are both read from your own clock, so the time between them needs no synchronization. Start there. If that round trip is small and orders still feel slow, look at your own software. If it is large, first check that your program was not paused or busy when the reply arrived; if it was not, the time goes outside your software.

Then look at the slowest orders. Sort a week of orders by round trip and note the time that only one order in a hundred exceeds; do the same for the first minutes after the open and around scheduled news. An average hides the moments traders complain about.

What can slow execution at a small prop firm?

Check these six causes first.

• The broker's API goes through a program you have to run. Interactive Brokers, for example, describes its TWS API as based on “connectivity to the Trader Workstation or IB Gateway”, so every order passes through one of those programs first. Its documentation sets a default limit of “50 requests per second” per client connection and warns that in some cases, above that rate, “some orders may be queued and delayed”. For that case, Interactive Brokers suggests switching to its FIX API. If this is your bottleneck, ask the broker how else you can connect
• The server is far from where the orders go. An office machine or a distant cloud region pays for the distance twice on every order, out and back, and no code change removes it. For the shortest distance, Nasdaq offers customers the opportunity to “co-locate their servers and equipment within the Nasdaq Data Center”. Before paying for colocation, measure the network round trip from your server to the broker's entry point
• Slow work runs before the order is sent. Writing the order to a database, waiting for a log line to reach the disk or asking another service whether the trade is allowed adds a wait to every order. When the database or the disk is busy, the wait grows. Keep what the order needs in memory and write records after the order has gone
• Network settings hold small messages back. An order is a small message. The Linux manual says that unless a socket option called TCP_NODELAY is set, outgoing data is buffered “until there is a sufficient amount to send out”. Your developers can check whether it is set
• The program pauses. Some garbage collectors stop the whole program while they clean up memory. Even Go's garbage collector, which does most of its work while the program runs, has “brief stop-the-world pauses”, and the Go garbage collector guide lists them among possible sources of latency. If a pause comes while an order is going out, the order leaves late. Charts, backtests or reports running on the same machine have a similar effect, because the order waits for the processor
• The broker's own path is slow. Its gateway, checks and routing sit in every order's path, and you cannot see inside. You can ask where its entry point is, which connection types it offers, which message limits apply to your account, and whether it will share its own timestamps for your orders

How is each cause fixed, and how big is the job?

• Decided to sent is large on every order. Take slow work off the order path and check the network settings. The job is a change in your code, sometimes a single setting
• Decided to sent jumps at busy moments. Find what the order waits for: a pause, a queue, a shared machine. The job is a code or hosting change, or a rebuild of the order path if the design itself queues
• Sent to acknowledged is large on every order. Move the server closer to the broker's entry point, or change the connection type. Expect a hosting contract and a move, or integration work for a new connection
• Sent to acknowledged jumps with volume. Find the limit you hit, at the broker or in your own connection. It may take only a conversation with the broker, or fewer messages from your side
• Acknowledged to filled is large. Look at the order type and the market. This is not an engineering job

Fix the cheapest confirmed cause first and leave a rebuild for last. Do not rewrite the system or switch brokers before anyone has timed an order, because the delay may be somewhere else.

Who can help us fix slow order execution?

Who can help depends on where the time goes.

• Your broker. The only party that can see and change its own side. Ask for its timestamps for your orders, its limits and its connection options
• A hosting or colocation provider, or the exchange's connectivity team. They rent space close to the broker's or the exchange's entry point and sell the network lines to it
• The vendor of your trading platform, if you trade through one you licensed. Only the vendor can change its internals, so bring it your timestamps
• An engineering company that works on trading systems. It measures the whole path, then changes or rebuilds the parts on your side, such as the order path, the risk checks and the connection to the broker
• Your own developers, if you have them. With the four timestamps, a capable developer can check every cause on your side of the path

Whoever you hire, ask four questions first:

• Will you measure before you propose a fix, and what exactly will you timestamp?
• Will the report separate our side, the wire and the broker, and show the slowest orders, not only the average?
• For any number you quote: at which percentile, under what load, on what date?
• If the delay turns out to be at the broker, what will you tell us?

If the answer to the last question is still a rebuild of your system, keep looking.

Where does amBrain fit?

amBrain is a Yerevan, Armenia software engineering company building low latency trading platforms, matching engines, and real-time bidding systems in Rust. amBrain diagnoses slow systems in trading, betting and ad tech: the running platform is measured end to end and the report names where the time goes.

amBrain builds algorithmic trading infrastructure: order execution, market data and pre-trade risk controls. Its trading work includes trading terminal development, order management systems, and FIX protocol exchange integration. amBrain takes over projects that stalled with another team and brings them to production. Three formats: full delivery, a dedicated team, or engineers embedded in your team.

If you are at the start, record the four timestamps on a normal day and on a busy one. Take them to whoever you call, amBrain or anyone else, so the first conversation starts from where the time goes.

Common questions

• Will rewriting our system in Rust make execution faster? Only if the time is lost inside your software, and only in the part the order passes through. Rust makes memory safety guarantees “without needing a garbage collector”, which removes garbage-collection pauses as a cause. It does nothing about a busy machine, distance or the broker
• Can we measure without changing our code? Yes, if your system already logs outgoing orders and incoming confirmations with times: the round trip from sent to acknowledged is in those logs
• Will faster execution get us better prices? Nobody can promise that. Speed shortens the time between a decision and the order's arrival; the price you get also depends on liquidity, the order type and what the market does in the meantime