Prayog: architecture¶

Diagram¶

flowchart LR
  subgraph Clients
    B[Browser: React terminal]
    BOT[Bots: Python SDK]
    SIM[Simulated traders]
  end
  T[Traefik] --> GW
  B & BOT & SIM --> T
  KC[Keycloak] -. JWKS, checked offline .-> GW
  subgraph EX[exchange-app: one JVM]
    GW[Gateway: REST + WebSocket, rate limits] --> RING[[Disruptor ring]]
    RING --> M[Matching: one thread]
    M --> J[Journal: SBE, CRC32C, fsync]
    J --> O[Outbound: answer client, market data, private updates]
    J --> SNAP[Snapshots and archiving]
  end
  O --> B & BOT & SIM
  J -. tail event log .-> KP[Kafka publisher, file checkpoint]
  KP --> K[(Kafka KRaft)]
  K --> PT[post-trade: ledger + leaderboard]
  PT --> PG[(PostgreSQL)]
  PT --> R[(Redis)]
  EX & PT --> PR[Prometheus] --> G[Grafana]

Walk-through script (about 3 minutes)¶

An order arrives. Traefik routes /api to the exchange. The gateway checks the JWT offline against Keycloak's keys, derives the account id from the token (SHA-256 of subject and label), applies the tier's token bucket, and publishes the command to the Disruptor ring with the pending HTTP response attached to the slot.
Matching. One thread applies the command to the order book: price-time priority, price bands, self-trade prevention, sessions. It reads time only from the simulated clock in the command stream. It emits events (accepted, trade, cancelled) into the slot.
Journal. The next stage writes the input and its events in SBE with a CRC32C, and fsyncs at the end of each batch (group commit: one fsync covers hundreds of commands at high rates).
Answer. The outbound stage completes the HTTP response with exactly that command's events, publishes book deltas and trades per symbol with sequence numbers, and sends private fills. Everything it says is already durable.
After the trade. A separate thread tails the event log on disk and publishes to Kafka with a file checkpoint. Post-trade consumes batches into Postgres in one transaction each, skipping duplicates by event id, then updates the Redis leaderboard.
Restart. Load the newest verified snapshot, replay the input journal after it, and compare every replayed event with the event log byte for byte. Any difference and the exchange refuses to start.

Where to point in the code¶

Part	Path
Matching rules	`services/exchange/exchange-core/.../core/MatchingEngine.java`
Book structure	`.../core/OrderBook.java`, `PriceLevel.java`, `RestingOrder.java`
Pipeline	`.../core/pipeline/`
Journal and replay	`.../core/journal/`
Outbound and market data	`services/exchange/exchange-app/.../app/core/OutboundHandler.java`, `.../app/market/MarketHub.java`
Kafka publisher	`.../app/kafka/`
Ledger	`services/post-trade/.../posttrade/ledger/`
Rules as scenarios	`exchange-core` test resources (Cucumber)

Paths are in github.com/rj8228/prayog.

Drill: redraw it from memory¶

On paper, without looking: clients → Traefik → gateway → ring → matching → journal → outbound; journal → snapshots; journal → Kafka publisher → Kafka → post-trade → Postgres + Redis; Prometheus → Grafana. Then explain each arrow in one sentence.

Follow-ups¶

What exactly happens between the order arriving and the client getting a response?

Gateway validates and authenticates (JWT checked offline against cached JWKS), applies the rate limit, claims a ring slot and writes the command plus the pending response into it, publishes the sequence. The matching thread applies the command and writes events into the slot. The journal thread encodes input + events in SBE with a CRC32C, appends, and fsyncs at the end of the batch. The outbound thread completes the HTTP response with that slot's events and publishes market data. Answer only after fsync.

Why does the gateway use WebFlux (reactive)?

The response is completed asynchronously by another thread (outbound) after the journal fsync. With a non-blocking server, the request thread is free while waiting; with a thread-per-request server you'd block a thread per in-flight order for ~12 ms. WebFlux also handles WebSockets naturally.

How does market data avoid touching engine state?

Market data is derived from the event stream (OrderTracker), never read from the engine's book. Only the matching thread ever touches engine state, so no locks. A property test checks that the derived book equals the engine's.

How is the account id derived, and why?

First 63 bits of SHA-256(subject/label) from the token. No account table to look up on the order path, stable across restarts and replays. Cost: readable names must be recorded separately (failure 6).

How does restart work?

Load the newest snapshot that passes its CRC, replay the input journal after it, and compare every replayed event with the recorded event log byte for byte. Any mismatch: refuse to start (better down than wrong). A torn last record (crash mid-write) is cut off, because it was never fsynced so no client was told about it.

What is group commit?

Instead of one fsync per command, the journal stage writes every command available in the batch, then fsyncs once. Under load, one fsync covers hundreds of commands, so throughput scales while latency stays at the fsync floor. The Disruptor's batching gives this for free (endOfBatch flag).

What does the post-trade ledger do in one transaction?

For each consumed batch: skip events whose id is already applied, insert fills, update positions and P&L, record the applied ids, commit. If anything fails, the whole batch retries; a redelivered batch is a no-op. After commit, update the Redis sorted set for the leaderboard; Redis is a cache rebuilt from Postgres if lost.