Market-data feeds
How an exchange tells thousands of clients what its order book looks like, many times a second, so that every client's copy is exactly right, and how a client knows when it isn't.
1. The problem
The order book changes with almost every order. Sending the whole book on every change would waste bandwidth; sending only changes is efficient but fragile: miss one and your copy is wrong forever, silently. Traders act on that copy. A wrong best price means wrong trades.
So a feed has to be cheap (only changes), exact (nothing applied twice or skipped) and self-checking (a missed message is noticed at once).
2. A business example: the cricket scoreboard
A radio commentator doesn't read out the full scorecard after every ball. When you tune in they give you the score ("India 142 for 3 after 20 overs"); after that they only say what changed ("four runs", "wicket"). Each ball is numbered: over 20, ball 3. If you hear ball 5 right after ball 3, you know you missed one and you wait for the next full score update instead of guessing.
Full score = snapshot. Each ball = delta. Over.ball = sequence number.
3. Level 1, 2 and 3 data
| Level | Contains | Used for |
|---|---|---|
| Level 1 (top of book) | Best bid, best ask, last trade | Tickers, simple displays |
| Level 2 (market by price) | Total quantity and order count at each price level | Depth ladders, most trading bots. What Prayog publishes. |
| Level 3 (market by order) | Every individual order, with its id and queue position | Strategies that model their place in the queue |
4. Snapshot plus deltas
- On connect, the client gets a snapshot: the whole book for each symbol, stamped with a sequence number
seq = N. - Then only deltas: "level 1,507.65 on the ask side is now 40 shares, 1 order" (0 shares = level gone), and trades. Each is stamped
N+1,N+2... - The client applies each delta to its copy. As long as no number is skipped, its book equals the exchange's.
The hard part on the exchange side is the hand-over: the snapshot and the first delta a new client receives must be consecutive. If a change slips in between taking the snapshot and registering the client, that client misses it forever.
5. Sequence numbers: noticing what you missed
Each message carries seq, one more than the previous message for that symbol. The client's rule is simple: if seq != last + 1, stop trusting the book and resynchronise. There are two ways:
- Re-snapshot: reconnect or ask for a new snapshot, then continue from there. Simple; Prayog does this.
- Retransmission: ask a recovery server for exactly the missing numbers. Faster for big books; used on UDP feeds where loss is normal.
6. Try it: drop a message
The exchange's book changes; deltas stream to a client. Press "Drop next message" to lose one in transit and watch the client notice and resync.
Exchange book
Client copy
7. Slow consumers
One client on a slow network can't read as fast as the market changes. The exchange's options:
| Option | Effect |
|---|---|
| Buffer without limit | Memory grows until the exchange falls over. Never. |
| Slow everyone down | One bad client hurts all. Never. |
| Disconnect after a bounded buffer | The client reconnects and gets a fresh snapshot. Prayog: 10,000 messages. |
| Conflate | Send only the latest state per level, skipping intermediate changes. Common for displays. |
8. How real exchanges do it
High-volume venues typically publish binary messages (often SBE or venue-specific formats) over UDP multicast: one packet reaches every subscriber, with no per-client cost. Packets can be lost, so they publish two identical copies over separate networks (A and B feeds; the client takes whichever arrives first) and run recovery services for retransmission and snapshots. Nasdaq's ITCH protocol over MoldUDP64 is a well-documented public example. Over the internet, retail-facing feeds use TCP or WebSocket, as Prayog does.
9. In Prayog
| Piece | Where |
|---|---|
| Book derived from events, after the journal | OrderTracker (exchange-core): property-tested to equal the engine's book after every command |
| Only whole commands' changes are published | An incoming order passes through states that never really existed; only the state between commands is sent |
| Snapshot and first delta consecutive | MarketHub: subscribing and publishing share one lock |
Per-symbol seq, gap detection | Python LocalBook raises SequenceGap; the web app marks the book "resyncing" and reconnects |
| Slow consumers | Disconnected after 10,000 queued messages |
| Proof | make e2e rebuilds every book from the live feed and requires it to equal the REST snapshot at the same seq |
10. Questions
Why not send the book only, every second?
Traders would see changes up to a second late and miss everything in between, including trades. Deltas give every change as it happens; periodic snapshots are a recovery tool, not the main feed.
Why number per symbol rather than one global number?
A client watching only INFY would otherwise see gaps whenever TCS changed. Per-symbol numbers let each subscription check itself.
Why derive the book from events instead of reading the engine's?
The engine belongs to one thread (the single writer). Reading its book from another thread would need locks on the order path. Deriving it from the events the engine already emits keeps the engine untouched, and the property test proves the two never differ.