How Chronicle Queue Transforms Big Data Solutions

What sets Chronicle Queue apart is that it’s a very lightweight message queue, a simple queue that can buffer vast volumes of queue data for hours, weeks, or even years without pushing back on the producer. It’s not limited by heap size, or memory size, only by the available disk space, so the average queue wait duration is determined solely by your storage strategy rather than artificial software limits.

For example, one of our users has recorded every tick of the Options Price Reporting Authority (OPRA) feed for eight years, allowing them to reprocess, replay, or remove any element queue from any point in time. The OPRA feed now peaks at 120 million messages per second, demonstrating Chronicle Queue’s suitability for real-time queue solutions at true Big Data scale.

In this article, we explore the world of Chronicle Queue to understand how it revolutionises Big Data solutions and look at what distinguishes it from other data messaging systems and message queues used in cloud services and on-prem deployments alike. We will also touch on how its underlying queue data structure, inspired by a linked list of off-heap memory-mapped files, helps producers and consumers share data safely with microsecond latency.

Image 1. Simplified Network diagram, with the messaging system Chronicle Queue

What is Chronicle Queue?

Chronicle Queue is a persisted journal of queue message entries that can handle concurrent writers and readers across multiple JVMs on the same machine. Unlike many other message queues, Chronicle Queue retains every message, which has several advantages. For example, messages can be replayed as many times as needed, making it ideal for debugging, for reproducing edge-case bugs, or for answering the classic “answer question” of what really happened in production. It also reduces the need for extensive logging, significantly speeding up your applications because the queue itself becomes the primary source of truth.

To illustrate this with a real-world example, a client’s trading system initially achieved an average latency of 35 microseconds without using Chronicle Queue. However, switching to Chronicle Queue, reduced their latency by 34 %, resulting in an impressive average latency of just 23 microseconds. This significant improvement highlights Chronicle Queue’s ability to dramatically enhance system performance, particularly in latency-sensitive producer-consumer share patterns such as those found in high-frequency trading systems.

What Sets Chronicle Queue Apart?

1. Speed: Chronicle Queue is built to handle hundreds of thousands of messages per second, even supporting multi-second bursts into the millions of messages per second. Some users have reported handling bursts of 24 million messages per second with a cluster of six servers, all while maintaining a near-constant average queue wait duration for downstream services.

2. No Flow Control: Unlike many other messaging solutions, Chronicle Queue operates without flow control between producers and consumers; a “producer-centric” solution that keeps the queue producer running at full speed even when consumers slow down.

3. Producer-Centric Solution: Chronicle Queue’s producer-centric approach ensures that the producer is never slowed down by a slow consumer. This has various advantages, including deterministic bug reproduction, independent microservice testing, and seamless service restarts and upgrades where an element queue can be removed or replayed without data loss.

4. Multi-language Offering: Chronicle Queue is now offered in multiple languages, including Java, Python, C and Rust, as well as a Proof of Concept in Go, which ensures efficient interoperability and the ability to leverage the strength of each language when designing a product design that spans heterogeneous stacks and cloud services.

Java for Low Latency?

Chronicle Queue leverages Java’s abstraction layer and protection to achieve impressive performance. It supports data structure sharing in memory for multiple JVMs, eliminating the need for TCP-based data sharing and drastically reducing time queue overheads found in conventional network message queues. The end result is a real-time queue producer consumer paradigm that delivers microsecond latency without resorting to unsafe native code.

Minimising Garbage

For ultra-latency-sensitive systems, minimising garbage collection is crucial. Chronicle Queue offers strategies to achieve this, such as keeping the allocation rate below 300 KB/s and translating between on-heap and native memory without generating temporary objects. Object pooling and support for reading into mutable objects further contribute to garbage reduction, helping maintain a predictable average queue wait and ensuring that time-critical message queues remain stable under peak load.

Conclusion

Chronicle Queue is an enabling technology for high-performance, Big Data applications with microsecond processing times. Its unique features, producer-centric speed, and garbage reduction strategies make it a top choice when performance is critical, whether you require a simple queue for in-house analytics or a sophisticated queue data structure for enterprise-grade trading platforms.