Off-heap memory, buffers and flyweights
Low-latency Java often works with raw bytes instead of objects: network packets, journal files, binary messages. This page explains Java's memory areas, the APIs for raw memory (from ByteBuffer to Unsafe to the new Foreign Memory API), the flyweight pattern behind SBE, and why Prayog's build needs an --add-exports flag.
1. The problem
Java objects are convenient, but they come with costs a trading system notices:
- Overhead: every object has a header, and references point all over memory, which is bad for CPU caches.
- Garbage: creating an object per message means the garbage collector has work to do, and its pauses land on someone's order.
- I/O copies: the OS can't read a file or socket directly into a heap object, because the GC may move it. Data gets copied through a native buffer first.
Working with flat byte buffers, ideally outside the heap, avoids all three.
2. A business example: the managed storeroom vs. a rented warehouse
The office storeroom is run by facilities (the garbage collector). You ask for a shelf, they find one, and every so often they close the room to rearrange it (a GC pause). Easy, but you don't control when you're locked out.
A rented warehouse (off-heap memory) you manage yourself. You know every shelf's address, nobody rearranges it, and trucks (the OS doing I/O) can unload straight onto it. But if you forget to clear a shelf, nobody does it for you.
3. What an object costs
On a 64-bit HotSpot JVM with compressed pointers (the default under 32 GB of heap), every object has a 12-byte header, and sizes round up to 8 bytes:
| Thing | Size | Note |
|---|---|---|
long in an array or field | 8 bytes | The data itself |
Long (boxed) | 16 bytes + a 4-byte reference to it | 2.5× the data, plus a pointer chase |
A TreeMap entry | about 40 bytes | Key, value, left, right, parent references and a colour flag |
A 4-character String | about 48 bytes | The String object plus its byte array |
| A whole NewOrder in SBE (client id "c1", symbol "INFY") | 48 bytes | One flat run of bytes, no references |
Prayog's order book still uses TreeMap<Long, PriceLevel>: textbook, clear and correct. Whether flatter structures are worth it is a question for the S9 benchmarks, not a guess.
4. Heap, direct and mapped buffers
Heap byte[] / ByteBuffer.allocate | Direct ByteBuffer.allocateDirect | Mapped FileChannel.map | |
|---|---|---|---|
| Lives in | Java heap | Native memory | The OS page cache, mapped into the process |
| Moved by GC | Yes | No | No |
| I/O | Copied through a temporary direct buffer | The OS reads and writes it directly | No read or write calls at all: touching memory is reading the file |
| Allocation | Fast | Slow (allocate once, reuse) | A system call; then page faults on first touch |
| Freed | By GC | When the buffer object is collected | When the buffer object is collected (no explicit unmap before the FFM API) |
| Size limit | An int index: 2 GiB per ByteBuffer. This is why Prayog's segments are capped well below that. | ||
5. Unsafe, VarHandle and the Foreign Memory API
| API | Since | Status | Notes |
|---|---|---|---|
ByteBuffer | Java 1.4 | Supported | Bounds-checked; carries mutable position and limit, which makes sharing awkward. Default byte order is big-endian. |
sun.misc.Unsafe | Early, never public | Memory-access methods deprecated for removal (JEP 471, Java 23) | Raw reads and writes at any address, CAS, fences. The engine of Netty, Agrona, the Disruptor 3.x and many more for two decades. |
VarHandle | Java 9 (JEP 193) | Supported | Typed access to fields, arrays and byte buffers, with memory-order modes (plain, opaque, release/acquire, volatile). The safe replacement for Unsafe's atomics. See the memory model. |
Foreign Function & Memory API (MemorySegment, Arena) | Final in Java 22 (JEP 454); preview in Java 21 | Supported | 64-bit sizes, deterministic freeing (closing an arena), bounds and lifetime checks, plus calling native code without JNI. The long-term home for off-heap work. |
Prayog targets Java 21, where FFM is still a preview, so it uses Agrona's buffers instead.
6. Agrona and the --add-exports flag
Agrona (by Real Logic, the authors of SBE and Aeron) is a library of low-latency building blocks: DirectBuffer (read), MutableDirectBuffer (write), UnsafeBuffer (wraps a byte[], a ByteBuffer or a raw address), ExpandableArrayBuffer, plus primitive collections and ring buffers. SBE's generated code reads and writes through these interfaces.
Agrona 2.x uses the JDK's internal jdk.internal.misc.Unsafe. Since Java 16 (JEP 396) and 17 (JEP 403), the module system strongly encapsulates JDK internals, so a library may only touch them if you say so on the command line:
--add-exports java.base/jdk.internal.misc=ALL-UNNAMED| Flag | Allows |
|---|---|
--add-exports module/package=target | Compiling against and calling public types of a non-exported package |
--add-opens module/package=target | Also deep reflection into private members (setAccessible(true)) |
ALL-UNNAMED means "all code on the classpath". Without the flag, the first Agrona buffer fails with IllegalAccessError ... does not export jdk.internal.misc, which is exactly what S8's first test run showed.
7. The flyweight pattern
A flyweight is one small, reusable object that gives a typed view over bytes it doesn't own. Instead of decoding 1,000 messages into 1,000 objects, you point one decoder at each message in turn:
NewOrderDecoder d = new NewOrderDecoder(); // made once
for (each record in the journal) {
d.wrapAndApplyHeader(buffer, offset, header); // re-point; no allocation
long price = d.price(); // reads 8 bytes at offset + 18
}RecordVisitor callback) after it moves on and you read someone else's data. Prayog's RecordVisitor documents "valid only during the call", and a codec instance is not thread-safe because its flyweights hold a position.The same idea appears all over Prayog: the Disruptor's pre-allocated CommandSlots are reused for every command, so a downstream handler must copy what it wants to keep.
8. Byte order (endianness)
A long is 8 bytes. Big-endian stores the most significant byte first (how we write numbers, and the "network byte order"); little-endian stores the least significant byte first (how x86 and ARM hold numbers in memory).
ByteBuffer defaults to big-endian, whatever the CPU. Writing with a default ByteBuffer and reading with a little-endian UnsafeBuffer turns 150,000 into a huge number. Prayog sets ByteOrder.LITTLE_ENDIAN explicitly on every buffer that touches the journal, to match the SBE schema.9. In Prayog
| Where | What |
|---|---|
FileJournal | A 1 MiB direct, little-endian write buffer, reused; grows only for an oversized record |
Segments.scan | Memory-maps each segment read-only and wraps it in an UnsafeBuffer for the codec |
JournalCodec | One flyweight encoder and decoder per message type, reused; one codec per thread |
JournalHandler | An ExpandableDirectByteBuffer reused for every command and event |
Parent pom.xml | agrona.jvm.args passes the --add-exports flag to Surefire and Failsafe |
10. Questions
Does off-heap mean no garbage collection at all?
No. The ByteBuffer object that owns the native memory is still a heap object, and the memory is freed when it's collected. The win is that the big data isn't scanned or moved, and that you allocate it once and reuse it.
Why are memory-mapped reads fast?
No system call per read and no copy: the file's pages in the page cache are mapped straight into the process. The first touch of each page costs a page fault; later touches are ordinary memory reads.
When should I reach for any of this?
Only after a benchmark says the simple version is too slow. Plain objects and collections are easier to read and to get right; Prayog keeps them on the engine's books until S9's numbers argue otherwise.