Learnings¶
What I learned building Prayog, session by session: the functional side (how exchanges work) and the technical side (how to build one). Each session file has:
- What was built: one paragraph, with links to the commit and the ADRs.
- Concepts: each one has a checkbox. I tick it once I can explain it without notes.
- In an interview: a one- or two-sentence answer I could give out loud.
The longer background reading lives in docs/overview/functional.html, docs/overview/technical.html and docs/overview/ring-buffer.html (with an interactive simulator). The decisions themselves live in docs/adr/.
Topic pages¶
Interactive explainers in docs/overview/, each with a business example, the mechanics, a try-it panel and how Prayog uses it. Open with open docs/overview/<name>.html.
| Area | Page | Covers |
|---|---|---|
| Trading infrastructure | ring-buffer.html | LMAX Disruptor, single writer, back-pressure (S7) |
| Trading infrastructure | sbe.html | Simple Binary Encoding: layout, flyweights, schema evolution, live encoder (S8) |
| Trading infrastructure | fix-protocol.html | FIX tag=value, order lifecycle, ExecutionReport, session layer and resend, message builder and parser |
| Trading infrastructure | journaling.html | fsync, write-ahead journal, group commit, torn writes, CRC32C, segments, replay, crash simulator (S8) |
| Platform | kafka.html | Kafka basics: log, topics, partitions, offsets, consumer groups, delivery guarantees, KRaft, listeners (S2) |
| Platform | oauth-oidc.html | OAuth2 and OpenID Connect with Keycloak: clients, PKCE, client credentials, JWT claims and validation (S2) |
| Platform | compose-traefik.html | Containers, Docker Compose, health checks, volumes, reverse proxying with Traefik (S2) |
| Trading infrastructure | market-data-feeds.html | Level 1/2/3, snapshot plus deltas, sequence gaps, slow consumers, multicast feeds; drop-a-message simulator (S11) |
| Trading infrastructure | market-making.html | Spread, inventory skew, adverse selection, self-trade and reconciliation traps; market-maker simulator (S13) |
| Core Java | java-memory-model.html | Happens-before, volatile, release/acquire, CAS, cache lines, false sharing, litmus test |
| Core Java | java-off-heap.html | Heap vs direct vs mapped buffers, Unsafe, VarHandle, FFM API, Agrona, --add-exports, endianness |
| Core Java | java-gc-jit.html | Allocation, collectors (G1, ZGC), garbage-free code, JIT tiers, warm-up, safepoints, JMH, coordinated omission |
| Core Java | modern-java.html | Records, sealed types, pattern-matching switch and exhaustiveness, virtual threads |
Index¶
| Session | File | Main topics |
|---|---|---|
| S1 Repo skeleton | S01-repo-skeleton.md | Module boundaries, Maven parent POMs, wrappers, lock files, pinning CI actions |
| S2 Local infrastructure | S02-local-infrastructure.md | Compose, health checks, Kafka KRaft and listeners, OAuth2/OIDC and PKCE, reverse proxy, secrets |
| S3–S4 Contracts and order book | S03-S04-contracts-and-order-book.md | JSON Schema contracts, integer money, price-time priority, order book data structures, determinism, property testing |
| S5 Market, cancel, modify | S05-market-cancel-modify.md | Commands and event sourcing, modify semantics, queue priority, information leaks, test coverage |
| S6 Bands, sessions, STP | S06-bands-sessions-stp.md | Price bands, session states, DAY expiry, self-trade prevention, kill switch |
| S7 Sequencer pipeline | S07-sequencer-pipeline.md | Single writer, ring buffers, back-pressure, wait strategies, simulated clock, testing concurrency |
| S8 Journal and replay | S08-journal-and-replay.md | Write-ahead journal, fsync and group commit, torn writes and CRC, SBE, deterministic replay checksums |
| S10–S13 Live market | S10-S13-live-market.md | Gateway, recovery, derived market data, feeds and gaps, account ids, market making, reconciliation, e2e correctness |
| Step 1.5 Admin, workspace, strategies | step-1.5-admin-workspace-strategies.md | Admin role and self-test, online replay, live simulation control, grid layouts, average-cost P&L, pure strategies, pre-trade risk |
| S14 Realism checks | S14-realism.md | Stylised facts, kurtosis, volatility clustering, sampling error, measuring from the feed |
| S15 Kafka publisher | S15-kafka-publisher.md | Topics, partitions and offsets, delivery guarantees, idempotent producers, contiguous acked checkpoint, journal as outbox, chaos tests |
| Duplicate client order IDs | step-2-duplicate-client-order-ids.md | Idempotency keys, retries after timeouts, bounded state, rule changes vs replay, test oracles |
| S16–S17 Post-trade and leaderboard | S16-S17-post-trade-and-leaderboard.md | Idempotent consumers, exactly-once effects, average cost in integers, zero-sum reconciliation, sorted sets, rebuildable views |
| Engine snapshots | step-2-engine-snapshots.md | Snapshot plus log, consistent cuts in a pipeline, atomic file replace, fallbacks, equivalence testing |
| S9 Benchmarks | S09-benchmarks.md | JMH, steady state, percentiles, coordinated omission, group commit, GC tails |
| S20 Rules as scenarios | S20-rules-as-scenarios.md | BDD, Gherkin, step definitions, scenario outlines, executable specifications |
| S21 Observability | S21-observability.md | Golden signals, counters vs gauges vs histograms, lag, dashboards as code |
| S18–S19 Terminal, blotter, ops page | S18-S19-terminal-blotter-ops.md | Official vs estimated P&L, polling vs push, role design, compressed sim time |
| Journal archiving | step-2-journal-archiving.md | Segmented logs, retention vs archiving, safe points, crash-safe file replacement, memory in background jobs |
| Allocation and GC | step-3-allocation-and-gc.md | JFR allocation sampling, boxing, open addressing, backward-shift deletion, GC pauses vs tail, interleaved A/B |
| S22 Docs and demo | S22-docs-and-demo.md | Failure-mode tables as test indexes, diagrams as code, demo storytelling |
| Industry context | industry-context.md | Exchange vs trading firm, HFT components, what production engines do differently |
Themes so far¶
- Determinism is the backbone. Integer money, time as an input, single-threaded matching and no hash-map iteration all serve one goal: the same inputs always produce the same output. That makes replay, crash recovery and later hot standbys possible.
- Fairness rules are anti-gaming rules. Price-time priority, the modify priority rules and opaque errors all exist to stop one participant from exploiting another.
- Tests must be checked too. Planting bugs on purpose (mutation testing) showed that one random test generator was missing a whole code path.