SBE: Simple Binary Encoding

The binary message format of the FIX Trading Community, used where every microsecond counts: exchange order entry, market data feeds and journals. Prayog uses it for its journal. First the idea, then the bytes, then how Prayog uses it.

1. The problem

Every order Prayog receives is written to disk before anyone is told about it. That means turning a Java object into bytes (encoding) and, on replay, bytes back into an object (decoding), millions of times a day. The format needs four properties:

Fast: no text parsing, no searching for field names, no allocation on the hot path.
Byte-stable: the same message must always give exactly the same bytes, because replay compares SHA-256 checksums of the bytes.
Versioned: next year's code must still read this year's journal.
Standard: something an engineer from any trading firm would recognise.

Text formats like JSON fail the first two. SBE was designed for exactly this list.

2. A business example: the pre-printed form

Imagine two ways a bank branch could send account-opening requests to head office.

A letter ("Dear Sir, please open an account for Mr. Rao, age 34..."). Flexible, but the clerk must read every word to find the age. That's JSON or FIX tag=value.
A pre-printed form where box 3 is always the age and box 5 is always the branch code. The clerk's eyes jump straight to box 3. Nobody reads the rest. That's SBE.

The form has one rule that matters later: you can add new boxes at the bottom, but never move or reuse an old one. Old clerks ignore the new boxes and keep working.

3. How SBE works

Write an XML schema listing messages, their fields and their types (int64, uint8, enums, strings).
Generate code. sbe-tool turns the schema into Java classes, one encoder and one decoder per message, at build time.
Encode in place. An encoder is a flyweight: a tiny reusable object that "wraps" a buffer at an offset and writes each field straight to its fixed position. No intermediate object, no garbage.
Decode in place. A decoder wraps the bytes and reads a field only when asked, by jumping to its offset. Reading the price of an order never touches the symbol.

The layout of a message

Header: blockLength (size of the fixed block), templateId (which message this is), schemaId and version. The header makes every message self-describing.
Fixed block: numbers and enums, each at a fixed offset. This is why decoding is a single memory read per field.
Groups: a repeated block, such as the list of instruments in Prayog's setup record.
Variable data: strings, as a length followed by the UTF-8 bytes. They come last and must be read in schema order, because each one starts where the previous one ended.

Byte order: Prayog's schema is little-endian, as SBE recommends. x86 and ARM are both little-endian, so encoding is a plain memory store with no byte swapping.

4. Try it: encode a NewOrder

This reproduces Prayog's real NewOrder layout from journal-schema.xml. Change a field and watch which bytes move. Notice that a longer client order id never shifts the price.

accountId side orderType price (paise) quantity clientOrderId symbol

headeraccountIdside, type pricequantitystrings (length + bytes)

With the default values, the bytes above are exactly the ones pinned by JournalCodecTest.layoutIsPinned.

5. Changing a schema safely

A journal written today must replay next year. SBE supports that with a few rules:

Change	Safe?	Why
Add a field at the end of the fixed block, bump the version	Yes	Old decoders use `blockLength` from the header to skip what they don't know. New decoders see an old version and return a null value for the new field (`sinceVersion`).
Add a new message type	Yes	New template id; old code never sees it in old journals.
Reorder or remove a field	No	Every later offset moves. Old bytes decode into the wrong fields.
Renumber an enum value	No	Old journals hold the old number. "SELL" would read as something else.
Reuse a template id	No	Old records would decode as the new message.

This is why Prayog's schema numbers its enum values explicitly instead of trusting the Java enum's order. If someone reorders RejectReason in Java, the bytes don't change.

6. SBE vs JSON vs Protobuf vs FIX tag=value

	JSON	Protobuf	FIX tag=value	SBE
Form	Text, named keys	Binary, tag + varint	Text, `tag=value` pairs	Binary, fixed offsets
Read one field	Parse everything	Scan tags	Scan tags	One memory read
Allocation	Many objects	A message object	Strings	None (flyweights)
Same message, same bytes	Not guaranteed	Not guaranteed across versions	Field order can vary	Yes
Human readable	Yes	No	Mostly	No
Typical use	Web APIs, Kafka in Prayog	Microservices (gRPC)	Broker and exchange connectivity	Exchange order entry, market data, journals

SBE is not a replacement for FIX; it is one of FIX's encodings. The same FIX business messages (NewOrderSingle, ExecutionReport) can be carried as text or as SBE. See the FIX page. For example, CME Group publishes its market data (MDP 3.0) and its iLink 3 order entry in SBE, and the Aeron messaging library uses SBE for its own protocols.

7. SBE in Prayog

Where	What
`exchange-core/src/main/resources/sbe/journal-schema.xml`	6 commands (ids 1 to 6), 7 events (ids 20 to 26), the engine setup (id 40)
`exchange-core/pom.xml`	`exec-maven-plugin` runs `sbe-tool` during `generate-sources`; output lands in `target/generated-sources/sbe`
`journal/JournalCodec.java`	Maps records (`NewOrder`, `Trade`...) to the generated flyweights and back
`JournalCodecTest`	Round-trip properties for every message, the pinned bytes, enum coverage
Parent `pom.xml`	`--add-exports java.base/jdk.internal.misc=ALL-UNNAMED` for tests, because SBE's runtime (Agrona) uses the JDK's internal Unsafe. See off-heap memory.

Prayog's codec still builds a Java record per decoded message, so it is not fully garbage-free. The flyweights make that possible later if the S9 benchmarks show it matters.

8. Questions

Why must strings be read in order?

A string's position depends on the length of every string before it. The decoder keeps a running "limit" that advances as each one is read. Read the symbol before the client order id and you are reading the client order id's bytes as the symbol. One of S8's planted bugs did exactly this, and the round-trip test caught it.

If SBE is so fast, why does Kafka use JSON in Prayog?

Kafka carries events to other services (post-trade, leaderboard) written in other languages, where readability and JSON Schema validation matter more than microseconds. The journal is private to the engine and on the critical path. Different jobs, different formats.

What does "zero-copy" mean here?

The decoder reads fields directly out of the buffer the bytes arrived in (or the memory-mapped journal file). Nothing is copied into an intermediate object first.