Comparing Approaches to Durability in Low Latency Messaging Queues

A significant feature of Chronicle Queue Enterprise is support for TCP replication across multiple servers to ensure high availability of application infrastructure. Designed from day one for real time trading, multi region deployments, and other latency-sensitive distributed systems, Chronicle Queue Enterprise allows you to keep data chronicle queue persisted across sites while maintaining microsecond-level performance. Over the past decade I have generally held the view that replicating data to a secondary system is often faster, and more fault tolerant, than sync-ing to disk—provided the round-trip network delay isn’t high, thanks to quality networks, well-tuned load balancing, and co-located redundant servers that minimise message queue latency. This is the first time I have benchmarked that assumption with a realistic example covering everything from message delivery to data stored on two hosts.

Little’s Law and Why Latency Matters

In many cases, the assumption is that as long as throughput is high enough, the latency won’t be a problem. However, latency is often a key factor in why the throughput isn’t high enough, especially when you are dealing with systems message queues that must pass messages between JVMs at scale. Real-time compression, fast message queuing, and low queue latency all hinge on keeping delays to an absolute minimum.

Little’s law states “the long-term average number L of customers in a stationary system is equal to the long-term average effective arrival rate λ multiplied by the average time W that a customer spends in the system”.

In computer terminology, the level of concurrency or parallelism a system has to support must be at least the average throughput times the average latency, which in practice equates to the number of outstanding message message operations in flight inside your distributed systems message fabric.

To achieve a given throughput, the level of concurrency increases with the latency. This generally also increases the level of complexity, contention, and risk in order to achieve this concurrency. Many event-streaming architectures boast a high inherent parallelism; however, financial systems generally do not, limiting how much concurrency can be theoretically achieved. More moving parts inevitably increase the risk of failure as well, which is precisely why fault tolerance and fault-tolerant designs remain paramount.

Best of Both Worlds

While sync-ing to disk is normally seen as a requirement for message durability, it carries a cost in terms of performance, directly increasing latency, but indirectly reducing throughput. In the context of latency message queues, every additional millisecond spent waiting for a flush to SSD or spinning disk eats away at the latency budget and can have knock-on effects for downstream services. Acknowledged replication, especially across multi-region links, gives similar guarantees and (spoiler alert) is quicker, delivering better latency throughput characteristics for real time services.

From time to time, however, you might get messages—such as orders, trades, or payments—that are too large or too valuable to risk losing, particularly when the data stored exceeds a certain commercial threshold.

A pragmatic trade-off, therefore, would be to use acknowledged replication by default, but to sync to disk when some business risk threshold is reached, either in an individual message or an aggregation of messages. This hybrid model is perfectly aligned with distributed systems that must survive data-centre failures in region deployments while still delivering sub-millisecond queue latency. In this test, I explore the difference that sync-ing only 10 times per second could make to latency chronicle queue behaviour.

Balancing Technical and Commercial Risks

Many IT systems tend to treat technical risks separately from commercial ones. For example, Apache Kafka, Redis, or NATS Streaming have the option to periodically sync data to disk—say every 100 ms—without regard to message content or value. Unfortunately, you have no idea of the value of the messages that could be lost in that window, and in multi region scenarios that risk multiplies with cross-site lag.

However, you can achieve improved outcomes by aligning the technical solution with the commercial risks. For example, with the Chronicle stack the queue roll cycle and write batching can be coupled with configurable thresholds so that you can sync to disk based on the content. A large order or payment individually, or a large total cumulatively, can trigger a sync. Instead of a cap of 100 ms of unknown value, you can have a cap of $10 m (you can also cap by time), thereby delivering a more fault-tolerant, real time response to commercial risk.

Chronicle Queue has multiple ways of triggering a sync; however, the simplest is to call sync() on the ExcerptAppender to ensure the chronicle queue data is persisted. In Chronicle Services, writing a sync() event triggers a sync on the underlying queue, keeping a record of when it was performed. Downstream services can read messages and wait for this event if they need to know a sync was performed, ensuring consistent message delivery semantics in distributed systems.

Low Latency vs Durability Requirements

Low latency systems’ ‘need for speed’ usually trumps the need for reliability, so the fastest option available—often pure async message queuing—is usually chosen. Messages are usually kept as small as is reasonably possible, e.g. 40 – 256 bytes, which fits neatly inside cache lines managed by OpenHFT Chronicle Bytes memory-mapped files.

However, many financial systems have higher durability but lower latency speed requirements. The message sizes are also typically larger—for example 1 – 8 KB—and the expectations around queue latency versus data safety can shift depending on business context.

Benchmarked Scenario

Many systems support flushing or syncing to disk periodically; however, this is not based on the content of the messages. With Chronicle you can select the critical messages that have to be sync-ed to disk. There are several programmatic calls you can make to trigger or wait for a sync based on message content, perfectly complementing any Aeron sequencer or Redis NATS-based event streaming tier you may already have. Assuming our application has a small portion of messages that must be sync-ed to disk based on business requirements, we can sync only when those messages are written, thus preserving disk space and improving real time responsiveness.

In this benchmark, (1) a client publishes a 1 KB message to a server over TCP, (2) the server writes the data to a Chronicle Queue on disk—remember, Chron­icle Queue supports memory mapped files for zero-copy efficiency—either using async as msync(MS_ASYNC), sync as msync(MS_SYNC), or sync-ing 10 times/s based on a simulated business risk (your use case will vary), (3) data is replicated to a second server over TCP to enable multi region deployments and enhance fault tolerance, (4) data is acknowledged, (5) on the replica the data is also written to disk via async flush under the control of the OS, leveraging chronicle bytes mappedfile to minimise I/O overhead, (6) after step 2 & 4, a commit message is sent back to the client. In practical terms, this means the system passes messages JVMs to JVMs with deterministic low queue latency while keeping the chronicle queue persisted on both primary and secondary hosts.

Each case used fast machines (Ryzen 9 5950X) and a low-latency network, demonstrating that with proper tuning you can build fault tolerant latency message queues that rival in-memory solutions while still providing durability. The OS was tuned for isolating CPUs (https://access.redhat.com/solutions/480473) to reduce latency, however no additional optimisations were added. Slower machines, disk subsystems, and higher network latencies will inevitably add to the timings below, yet the architectural principles—openHFT Chronicle Queue, its openhft chronicle bytes storage layer, and the efficient use of memory-mapped files—remain valid. No tuning was made to optimise how sync performed, highlighting the out-of-the-box capability of java openhft chronicle libraries.

Low Latency Options with Small Messages

A simple way to reduce latencies is to do less work. Low latency systems tend to use smaller messages around 256 bytes; what latency can we get if we don’t need strong resilience guarantees? In each case, the same configuration is used; the difference is which points are timed. This chart illustrates the performance you can get if you consider:

• Time to publish only; this is the minimum you can do. All it does is write the data to a buffer so it can be written to the network asynchronously. The data doesn’t even reach step 1 in the diagram above, yet it is already safely in the openhft chronicle bytes buffer in the process’s memory.

• Time to wait for a response from Server 1 without an acknowledgment from a second server. This does steps 1, 2 & 6, reflecting a single-region deployment where message queuing overhead is minimal.

• Time to wait for a response from Server 1 after an acknowledgement; this does the same steps 1, 2, 3, 4, and 6 as before, mirroring a multi region replication path.

The chart below shows data from the same run, and the different lines represent which points we measure time at, to show the difference it makes if you wait for different stages of persistence and how that impacts message queue latency across distributed systems.

While these tests benefit from smaller messages (compare “M.2 Async&Ack” in the last chart below), the main speed improvement is not waiting for the same stage of persistence. What you see is that real time load balancing between nodes, along with openhft chronicle queue’s ability to stream data instantly, delivers near-wire performance even under pressure.

You can customise how your system behaves—either at the queue level, by message type, or even based on the message’s contents—to align the technical risks to the commercial risks. Chronicle Queue supports dynamic roll cycle adjustments so that daily, hourly, or even minute-level files can be created without wasting disk space, ensuring the chronicle queue data footprint stays lean.

Waiting for either Replication Acknowledgement OR a Sync to Disk

For systems where a sync to disk is currently required, there could be a significant latency improvement if the alternative is to wait for acknowledged replication. In this case, the latency is the same as the “Async&Ack” options (plus network round-trip time), falling back to the “Sync&Ack” latency when the replica isn’t available (or after failover to the secondary system). This strategy is particularly powerful in multi region environments where network fabrics are fast and predictable, and it complements existing message queues such as Redis NATS or Apache Kafka by providing deterministic low-latency persistence for critical paths.

SATA Solid State Drives with Medium Sized Messages

Above we compared the same configuration timed at different stages. In this case, the end-to-end is timed, with different options for sync-ing to disk.

In use cases requiring higher guarantees, the messages are often larger. In this test, 50K/s messages of 1 KB are timed from publishing on a client to receiving a committed message from the “Server 1”. The “SYNC 10/s” assumes that on average, a SYNC is required ten times a second. This can be based on the content of the messages—e.g. a large order or payment—rather than just periodically. By letting the java openhft library make this decision, you avoid needless stalls and gain headroom for peak loads.

Figure 2. 50K/s messages of 1KB timed from publishing on client to receiving a committed message

Note: the round-trip time for an acknowledgement from a second machine is much faster than a sync to disk. The cost is about the round-trip time for the network, which, on modern data-centre fabrics, can be as low as a few microseconds—ideal for real time applications that cannot afford queue latency spikes.

Solid State Drives are not unusual in enterprise-grade data storage systems. They perform much better than Hard Disk Drives; however, as you increase the throughput, they can be a bottleneck, especially when you have to guarantee that every byte is on stable storage before message delivery completes.

As you can see from the yellow line, this significantly reduces the typical latency while improving the whole latency distribution for this SSD. Chronicle queue persisted data, combined with async replication, demonstrates a compelling balance of durability and performance.

M.2 Solid State Drives with Medium Sized Messages

M.2 drives perform much better across the board, and for the drive tested, even at 200Kmsg/s outperformed the SSD. Nevertheless, selective sync-ing still significantly improves the typical latency and the high-end latencies. This is the same test as above but with a higher throughput of 200K/s, illustrating how latency data and latency throughput scale in modern NVMe architectures.

Again, you can see that the yellow line has significantly lower latencies with selective sync-ing compared to sync-ing every batch of messages. The openhft chronicle queue relies on memory mapped files to push bytes straight to the OS page cache, enabling real time read messages with minimal CPU cost while keeping your data safe.

Head Room

To put this in context, VISA has a capacity of 65,000 transaction messages per second (as of Aug 2017). This is a single pair of servers via a single TCP connection. Having greater headroom immediately available can increase reliability, reducing the risk of the system being overloaded in a burst of activity and reducing the time the system takes to return to normal operation. In large multi region deployments, where traffic can shift rapidly between data centres, such headroom is an essential ingredient in maintaining service-level objectives.

Increased head room reduces the risk of a cascading failure in the event of a burst of activity or an interruption. Chronicle Queue Enterprise, backed by openhft chronicle bytes and designed for fault tolerance from the ground up, makes it straightforward to build distributed systems that continue to deliver messages even under adverse conditions.

Conclusion

Sync-ing to disk with SSDs was under 25 milliseconds most of the time for even decent throughputs, up to 50K/s. However, using high-performance M.2 drives increases the throughput up to 200K/s and is still under 25 ms most of the time.

Typically, latency can be reduced significantly if the application can selectively sync data based on the contents of those messages—e.g. based on value. This can result in typical latencies close to just waiting for acknowledged replication, delivering the holy grail of low-latency messaging queues in real time production.

The fastest option tested was to async to disk and wait for acknowledged replication. The higher percentile latencies can be one-tenth or better using this strategy. When combined with the deterministic design of java openhft chronicle and its unparalleled support for distributed systems message flows, you achieve a solution that comfortably handles multi region deployments, minimises disk space usage via efficient roll cycles, and provides robust fault-tolerant message queuing without compromising on speed.