Benchmarking our Kdb+ Loader

As part of our partnership with AquaQ and integration into Kdb+ we have been carrying out some benchmarking details of the test environment are:-

– Producer writes messages to Chronicle Queue on Host 1

– Tailer reads messages from Queue on Host 1

– Tailer optionally batches messages

– Tailer then writes batches to Kdb+ on Host 2

– Runs for 100k messages at 10k msgs/s, and 300k messages at 30k msgs/s. 50k message warmup.

– Times are end-to-end: from when the producer writes the message to when the data has been written to Kdb+

Note we are including a network hop in the tests as – although slower – this is most likely how Kdb+ will be used in practice.

We can see that the environment saturated at a sustained rate of around 33k messages per second at which point messages remain in the Chronicle Queue until the adapter/Kdb+ can service them.

All timings are in Microseconds

More Information about the loader can be found here Chronicle Software integrates Chronicle Queue with kdb+ – Chronicle Software

The loader can be accessed here AquaQAnalytics/kdb-chronicle-queue: Java adaptor from Chronicle queue to kdb+ (github.com)