What is Chronicle Map
Chronicle Map is a high performance, off-heap, key-value, in memory, persisted data store. It works like a standard java map yet it automatically distributes data between processes, these processes can be both on the same server or across your network. In other words its a low latency, huge data key value store, which can store terabytes of data locally to your process.
It is able to access this data with sub-micro-second latency. Chronicle is built on Java but it does not undergo, the garbage collection pauses, which most Java applications experience. This is because chronicle stores your data off heap. Even if you have a large cache with lots of updates, the garbage collector will have almost no effect; resulting in more predictable latency and throughput. The off heap data is stored in a memory mapped file, which offers further resilience to unforeseen outages.
How it works
You define the structure of the entries you wish to store and set the maximum number of entries. Chronicle will create and appropriately sized memory mapped file, say 4TB. The size of the memory mapped file is not constrained by physical memory, so unlike heap based key value stores, chronicle eliminates the pauses that can occur when the store has to be resized.
If your data grid spans two or more processes on the same server, each process will share the same 4TB space, fine grain locking will ensure that your data integrity. The data is transferred directly using shared memory, when replicating on the same server we don’t use TCP/IP loopback, which will add additional latency, we just use memory. This approach gives you outstanding performance, we have measured data being transferred between processes in a staggering 40 nanoseconds.
Chronicle Map offers :
- High throughput, low latency, cross process, persisted key value store.
- Supports very high update rates by only replicating the latest value.
- Off heap data storage, with micro second latency.
- Designed to be used in Java, in a pause less, garbage free manner.
- Easy to adopt and open source based on the ConcurrentHashMap.
- Better utilizes your existing hardware by offering vertical scalability.
- UDP support for high fan out.
- TCP connectivity.
- Concurrent writers on one machine using one copy in memory.
- Concurrent multi-master writers across machines, with a reconciliation process.
- Multiple readers with TCP and UDP replication.
- Resilient writes ( if a process dies, the last write is not lost. )
- Entirely embedded, you don’t need any additional processes.
It is an ideal choice if :
- You require the latest value based on a key.
- Your update rate can exceeds the rate consumers can consume the data.
- You have limited Network bandwidth.
Chronicle Map has been tested to do
- 30 million updates per second with 500 million key-values on a 16 core machine, with a 64 MB heap and no minor collections.
- multi way multi-mastering
- 1.6 million updates per second replicated across a low bandwidth between two laptops.
- A ping-pong test between processes of 25 million per second. One process set a flag from false to true, the other from true to false.
- Data is not lost even if the JVM crashes on the next line.
Find out more…
Peter Lawrey talks about shared memory in java and Chronicle Map performance *
* Chronicle Map used to be called Shared Hash Map