Garbage collection and the JIT, for low latency

Java is fast once it's running hot, but two JVM mechanisms decide its worst moments: the garbage collector (pauses) and the just-in-time compiler (warm-up and deoptimisation). Trading systems care about the worst moments, so this page explains both, how to keep them off the order path, and how to measure latency without fooling yourself.

1. The problem: the average lies

A matching engine that answers in 20 µs on average but stalls for 50 ms once a minute is a problem: those stalls land on real orders, often during the busiest moments, when allocation is highest. That's why trading systems quote percentiles (p99, p99.9, max) rather than averages, and why the JVM's pause behaviour matters more than its raw speed.

2. A business example: the restaurant's cleaning crew and the new chef

• Garbage collection is the cleaning crew. The old way: close the restaurant every so often and clean everything (a stop-the-world pause). The modern way: clean while service continues, pausing only for seconds to swap trays (a concurrent collector). Best of all: make less mess (allocate less).
• The JIT is a new chef. On day one they follow the recipe card slowly (the interpreter). After cooking a dish a few thousand times they do it from memory, fast, with shortcuts (compiled code). If the menu changes, the shortcuts no longer apply and they go back to the card for a while (deoptimisation).

3. Allocation and garbage collection

• Allocation is cheap. Each thread gets a private chunk of the young generation (a TLAB, thread-local allocation buffer); new just bumps a pointer.
• Collection is the cost. Most objects die young (the generational hypothesis), so collectors clean the young generation often and cheaply, copying the few survivors. Pause time grows with the number of surviving objects, not dead ones.
• Escape analysis. If the JIT proves an object never leaves a method, it can skip allocating it entirely and keep its fields in registers (scalar replacement). Many short-lived iterators and small records disappear this way, but you can't rely on it without measuring.

4. Choosing a collector

Concurrent collectors aren't free: they take CPU from your threads and add small costs (barriers) to reads and writes of references. Low pause is a trade, not a gift.

5. Garbage-free hot paths

The most reliable way to avoid GC pauses is to give the collector nothing to do on the critical path. Common techniques:

6. The JIT and warm-up

1. Interpreter. Bytecode runs instruction by instruction, while the JVM counts calls and branches.
2. C1 (tiers 1 to 3). After some hundreds to thousands of calls, a quick compiler produces decent machine code with profiling built in.
3. C2 (tier 4). After roughly ten thousand calls, the optimising compiler uses the profile: it inlines hot callees, removes unneeded checks, unrolls loops and applies escape analysis.
4. Deoptimisation. C2 bets on what it saw ("this call always goes to NewOrder"). When a bet fails, the code is thrown away and the method runs slower until it's recompiled.

Consequences for a trading system:

• The first orders of the day are slow unless the code is warmed up first. Firms replay recorded traffic before the open, so the JIT has seen every path. Prayog's replay tool could do exactly that.
• Rare paths are cold. The kill switch or a session halt may run interpreted the first time it's needed.
• Call sites with many receiver types ("megamorphic", more than two seen types) are hard to inline. A switch over a sealed type (as in Prayog's engine and codec) compiles to a type check per case, which stays predictable.

7. Safepoints

Some JVM operations (most GC phases, deoptimisation, taking thread dumps) need every Java thread stopped at a known point, a safepoint. Threads only check for a pending safepoint at certain places (method returns, loop back-edges), so the pause includes "time to safepoint": waiting for the slowest thread to arrive. Logging with -Xlog:safepoint shows both parts. A long counted loop without a check can delay everyone.

8. Measuring latency honestly

• Microbenchmarks need JMH. A hand-written timing loop gets fooled by warm-up, dead-code elimination (the JIT removes work whose result you never use) and constant folding. JMH handles warm-up iterations, forks fresh JVMs and provides Blackhole to consume results. Prayog's exchange-bench module uses it in S9.
• Record full distributions. HdrHistogram records every latency into fixed-precision buckets cheaply, so p99.9 and max are exact, not averaged away.
• Avoid coordinated omission. A load generator that waits for each answer before sending the next stops sending during a stall. The stall then looks like one slow request instead of hundreds. Measure from when each request should have been sent (Gil Tene's term and fix, built into HdrHistogram).

9. Try it: tails and pauses

A service handles requests at a steady rate, each in about 20 µs. Every so often the JVM pauses; requests arriving during the pause wait until it ends. Compare the honest percentiles with those of a load tester that suffers from coordinated omission.

10. In Prayog

11. Questions

Why not just make the heap huge so GC never runs?

Young collections still happen as the young generation fills; a bigger heap mostly makes the rare full collection longer. Some firms do size the heap so that no old-generation collection happens during a trading day and restart nightly, combined with very low allocation. That's a deliberate design, not a default.

Is C++ still better for this?

C++ has no GC or JIT warm-up, so its worst case is easier to control. Java teams get close with garbage-free code, careful warm-up and modern collectors, and gain safety and productivity. Several well-known exchanges and trading firms run their matching engines on the JVM.

What does "mechanical sympathy" mean?

Writing software that suits how the hardware works: sequential memory access, few cache misses, little contention, no needless allocation. The ring buffer, SBE and the single-writer engine are all examples.