Prayog: decisions and numbers¶
The full records are the 23 ADRs in docs/adr/ of the public repo. This page is the short version: the choice, the
reason, the alternative I rejected, and the cost I accepted.
| # | Decision | Why | Rejected alternative | Cost accepted |
|---|---|---|---|---|
| 3 | Order book = TreeMap of price levels, orders in intrusive linked lists |
O(log n) best price, O(1) cancel; book depth barely changes latency (S9) | Array of price ticks (faster, but wastes memory over a wide band) | Boxed Long keys allocate on the hot path |
| 4 | One sealed Command type, one entry point apply(Command) |
Every input is journaled and replayed the same way | Separate methods per action | Every new feature is a new command and a codec change |
| 5 | Price band = whole percent of a static reference, edges rounded inward to the tick | Integer maths only, never floating point for money | Dynamic bands | Less realistic for fast moves |
| 6 | LMAX Disruptor pipeline: matching, then journal, then outbound; one writer thread | No locks in matching; batching for free | Locks around a shared book; actors | Pipeline lifecycle is subtle (see failures, story 2) |
| 6 | Simulated clock as an input | Same seed and inputs give a byte-identical event log | System time | Every time-based rule must take the clock as input |
| 7 | Journal in SBE with CRC32C; answer only after fsync | A client is never told something a crash could erase | Answer first, write later | Median latency equals fsync (12 ms on the Mac SSD) |
| 9 | Market data derived from events (OrderTracker), never read from the engine |
No thread but the matcher touches engine state | Readers lock the book | A second model of the book, checked by a property test |
| 9 | Account id = first 63 bits of SHA-256(subject/label) | No account table; stable across restarts and replays | A database sequence | Names must be recorded separately (failures, story 6) |
| 13 | Kafka publisher tails the event log on disk with a file checkpoint | Kafka off the order path; outage only delays the stream | Publish from a pipeline stage | Duplicates possible, so consumers dedupe by event id |
| 14 | Matching-rule changes are journaled commands (SetRules) |
History replays under the rules it was made with | Code version decides rules | Rules code must keep old versions |
| 15 | Post-trade: idempotent consumer, one transaction per batch, Redis leaderboard rebuilt from Postgres | Redelivery changes nothing; Redis is a cache | Ledger in the exchange | Another service and JVM |
| 16 | Snapshots are journaled points (TakeSnapshot), verified on restore |
Fast restart without trusting a snapshot blindly | Periodic dumps outside the journal | Snapshot work on the matching thread for a copy |
| 18 | Exchange rules as Cucumber scenarios | Rules readable by non-programmers, run in CI | Only unit tests | Another test layer to maintain |
| 19 | Prometheus and Grafana, one dashboard from order to ledger | Find tails like the GC story quickly | Logs only | Memory on an 8 GB laptop |
| 20 | Rate limits by tier from the token (people 20/s, bots 50/s, simulated traders 500/s) | A user's bot cannot starve people | One shared limit | Tier config per client |
| 22 | Gzip finished journal segments, keep every record | 2.6-4.8x less disk, history intact | Delete old segments | Full replays read archives 6-9x slower |
| 23 | Boxing-free order index (LongObjectMap: open addressing over long[] + Object[]) |
JFR showed HashMap<Long, RestingOrder> was a third of the matching thread's bytes |
Keep HashMap |
Custom collection code to maintain |
The three I would lead with¶
- Answer only after fsync, then replay to prove it. Durability and determinism are the backbone: recovery, the replay check and a future hot standby all depend on it.
- Kafka is off the order path. The publisher tails the durable log, so a Kafka outage delays the stream but never
blocks trading.
make e2estops the broker and checks every event id arrives. - Rules are versioned inputs. Learned the hard way (failure 1).
The numbers¶
Every number below comes from docs/benchmarks.md in the public repo, which records the command, machine and date.
Machine for all: Apple M1 (4 performance + 4 efficiency cores), 8 GB RAM, Java 21.0.6, an ordinary laptop.
The ones to say out loud¶
| Claim | Number | Source entry |
|---|---|---|
| Matching cost per command | about 60-120 ns on one thread | 2026-10-07 S9 JMH |
| Order path median with fsync | about 12 ms (the Mac SSD's fsync) | 2026-10-07 S9 HdrHistogram |
| Throughput with journal and fsync | 20,000 commands/s held (group commit) | same |
| Order path without the disk | 0.5 µs median (busy-spin), 24 µs (blocking) | same |
| fsync inside the Docker volume | 0.8 ms per 4 KiB O_DSYNC write |
2026-10-08 Docker tail |
| Live stack, steady state | p50 2.8 ms, p99 14 ms, p99.9 25 ms | same |
| Journal archive ratio | 2.6x (input), 4.8x (events) | 2026-10-08 archiving |
| Allocation cut by the boxing-free index | 12-21% fewer bytes per operation | 2026-10-08 ADR 0023 |
Matching engine (JMH, one thread, no journal)¶
| Operation | depth 10 | depth 1,000 | Bytes/op after ADR 0023 |
|---|---|---|---|
| Passive order joins best bid, then cancelled (2 commands) | 123 ± 25 ns | 142 ± 32 ns | 392 (was 496) |
| Aggressive buy fills, new ask refills (2 commands, 1 trade) | 237 ± 146 ns | 225 ± 56 ns | 576 (was 656) |
| Market order sweeps 3 levels (13 commands, 12 fills) | 2,047 ± 214 ns | 2,177 ± 472 ns | 4,158 (was 4,799) |
Why depth barely matters: best price through a TreeMap (O(log n)); orders in intrusive linked lists, so a cancel
never walks the queue.
Order path end to end (HdrHistogram, no coordinated omission)¶
| Run | p50 | p99 | p99.9 |
|---|---|---|---|
| 1,000/s, journal on | 12.0 ms | 35.0 ms | 71.0 ms |
| 20,000/s, journal on | 12.7 ms | 140.8 ms | 286.5 ms |
| 20,000/s, busy-spin, journal off, G1 | 0.5 µs | 13.5 ms | 118.7 ms |
| 20,000/s, busy-spin, journal off, ZGC | 0.5 µs | 1.3 ms | 6.9 ms |
GC (2026-10-08, host, 3 repeats each, journal off)¶
| Setting | Pauses per 25 s run | Longest pause |
|---|---|---|
| G1 default | 11 | 25-26 ms |
| G1, fixed 512 MiB heap, pre-touched | 6 | 10.6-11.9 ms |
| Parallel, 512 MiB | 1 | 12-13 ms |
| Generational ZGC | 0 stop-the-world | - |
The p99.9 tail varied up to 10x between repeats of one setting, so on this laptop only pause numbers are reliable.
The Docker tail story (in numbers)¶
| Window | Collector | p99 | Longest pause |
|---|---|---|---|
| Demo recording (host short of memory) | Serial | 410 ms - 1.41 s | 5.93 s full GC |
| Clean window | Serial | 14 ms | 44 ms |
| Clean window | Generational ZGC | 78 ms (allocation stalls) | 10 ms |
How to say the caveats¶
- "These are laptop numbers. The error bars are part of the result, and I interleave A/B runs because drift is bigger than most effects."
- "The engine is not the bottleneck; fsync is. On a server I would replicate the journal instead of waiting on the disk."