What is a Queue-backed Map

A QueueBackedMap implements java.util.Map API (actually java.util.concurrent.ConcurrentMap) and uses a Map internally to hold its data. For persistence (and replication), it uses a ChronicleQueue to journal changes to its entries.

The differences between ChronicleMap and QueueBackedMap are summarised in this table. QueueBackedMap can be backed either with an on-heap ConcurrentHashMap or an off-heap ChronicleMap.

QueueBackedMap

QueueBackedMap contains a Map. For persistence, every change made to the QueueBackedMap (for example, using put, remove, or update methods) is written as an event to the Chronicle queue that backs the map. These events can then be replicated from a source QueueBackedMap and replayed into sink QueueBackedMap s.

A QueueBackedMap can contain very large numbers of keys - it is constrained by Chronicle Map which has been used with billions of keys in production.

Once the QueueBackedMap has been created it can be used like a normal Map - you can get, put and remove.

Restarting a QueueBackedMap

When restarting a QueueBackedMap, events are replayed from the backing queue as per above diagram. There are 2 possible scenarios:

all events from the start are replayed. Replay of events happens very quickly - in our tests we see approximately half a million events replayed per second, although if the values stored in the Map are large, or are slow to serialise, this will be slower. We recommend to start with this scenario and test the QueueBackedMap with some representative data.

if QueueBackedMapBuilder.persistedMap() is used, the underlying ChronicleMap is stored on disk (in a memory-mapped file) and events are replayed not from the start, but from the last place that they seen when the QueueBackedMap was closed. This is a useful startup time optimisation if your queue has a lot of events in it, and you have a lot of keys.

Writing events to a QueueBackedMap

When changes are made to the QueueBackedMap, entries are written into the backing queue as above.

Periodic shrinking

As QueueBackedMap stores deltas in an ever-growing queue, there may be some periodic housekeeping required to trim the size of the underlying queue. There are two approaches that can be followed, detailed below.

Note

If the source queue is shrunk then it is a good idea to archive old roll cycles on any sink QueueBackedMap instances (make sure sink replicator is stopped first). An alternative is to remove the queue files on sink instances and they will be regenerated when the replication cluster is re-started.

Snapshot and archive

Calling QueueBackedMap.writeSnapshot will write a clear message followed by the contents of the QueueBackedMap to the end of the queue. Any historical data i.e. any roll cycles before the current roll cycle (current when QueueBackedMap.writeSnapshot was called) can be safely archived.

Snapshot to another queue

To write a minimal version of the QueueBackedMap to a new backing queue, call QueueBackedMap.writeSnapshot(<dest queue>). This will write a snapshot of the current QueueBackedMap state.

What if my persisted Chronicle Map becomes corrupted?

Chronicle Queue should never become corrupted as it makes sure to mutate its state with atomic operations.

If using a persisted map, there are some circumstances (e.g. power loss) where a Chronicle Map can become corrupted, and so it is recommended that if there is a possibility of the persisted Map becoming corrupted, to remove its associated file before re-opening the QueueBackedMap.

Benefits of a QueueBackedMap

One benefit of the QueueBackedMap is that it does not conflate messages. Every change to the map is captured, and written to the Chronicle queue. In replicating a QueueBackedMap, every change is replicated. This is unlike a ChronicleMap which conflates changes; only the last value is replicated and the intermediate values are conflated.