Unix has been around for more than 50 years, and the original design principles must be good enough for it (and its derivative, Linux) to be the most widely used Operating System on the planet – 80% of servers, most supercomputers, and the most deployed OS (Android). It is also the most popular OS on Mars!

In today’s real time financial markets, where frequency trading, algorithmic trading and ultra latency strategies dominate, the same reliability and performance virtues that made Unix ubiquitous are now vital to trading firms, market makers and latency traders competing in a millisecond environment. When every microsecond can influence market quality, price discovery and short term volatility, choosing an operating system rooted in efficiency is essential to reduce latency and respond to market events instantly.

Much of Unix’s success can be attributed to the “Unix Philosophy” which can be very briefly summarised as:

• Write programs that do one thing and do it well

• Write programs to work together

• Write programs to handle text streams, because that is a universal interface

Programs that do one thing are easy to understand, and simple to test. Key to the Unix philosophy is how modules can be composed – the Unix way is generally that all modules communicate with each other using a protocol that they can all understand – text, and various meta-protocols can be layered over the top of text e.g. columns and fields (separated by space, comma etc.), and connected together using pipes.

For trading frequency scenarios, this seamless composition allows proprietary algorithms to consume market data feeds, perform submission cancellation workflows, and publish orders without introducing increased latency activity that can degrade market quality measures.

The above simple rules allow a system to be composed of simple components, connected together, and for significantly complex application behaviour to emerge as a result. The canonical example of the power of the Unix Philosophy is the famous Knuth vs McIlroy competition to build a word count program; McIlroy builds a six command shell pipeline that is a complete (and bug-free) solution to the problem.

In a similar vein, frequency traders often stitch together specialised micro-utilities to parse limit book updates, measure latency activity and react to market events millisecond by millisecond, proving that modular design enhances agility under dynamic market conditions.

The Unix philosophy in Enterprise IT

The above has arguably never really translated to Enterprise IT – an Enterprise application tends to deal with relatively complex problems, be made up of modules with a greater scope and business functionality, and despite numerous attempts over the years to try and come up with a high performance standard for connection of modules together (COM, CORBA, SOAP, JSON/REST/HTTP anyone?), an effective standardised connection mechanism has never “stuck”.

For market structure equities or derivatives platforms, the lack of a uniform, low-overhead messaging fabric can directly impact frequency trading performance, create latency trading market inefficiencies, and even exacerbate declining prices heightened by short term volatility.

The Chronicle solution

Chronicle’s solution for the Unix Philosophy in Enterprise IT involves composing systems from

• Programs that do one thing and do it well – single-threaded Java¹ microservices that align perfectly with prop trading firms seeking deterministic behaviour in a millisecond environment

• Connected together with a mechanism to transport structured and self-describing data – Chronicle Queue plus Chronicle Wire Method Readers and Writers, proven across multiple hft dataset benchmarks and nasdaq hft dataset replays

These open source technologies not only provide the benefits of Unix tools plus pipes, but also

• Are low latency and low garbage, and thus are suitable for building systems that require high throughput, microsecond response times and predictable latencies. This directly helps reduce latency for latency trading strategies and improves overall market quality.

• Persist all data that is sent between modules, facilitating debugging, troubleshooting and out-of-band reporting, while giving market participants a reliable audit trail that can be replayed to analyse impact latency activity after major market events.

• Allow individual modules to be stopped/restarted/upgraded without interrupting others, which is critical when proprietary trading desks need to hot-swap proprietary algorithms as market conditions shift.

Example

Below is some example code that is a super-simplified version of a workflow that is common in the world of financial markets (most of Chronicle’s users are in this space):

In production, similar pipelines are used by frequency trading firms to aggregate market data, build depth limit book views, and execute strategies that respond market events at lightning speed.

In this example

1. an exchange emits a lot of fast-moving price data which are sent to…

2. an aggregator which assembles price deltas into a “book” of prices which are sent to..

3. a strategy which decides whether an order should be sent to…

4. the market

Such a data path mirrors real time submission cancellation patterns observed in frequency trading, where market makers continuously refresh quotes to maintain tight spreads and preserve market quality.

The exchange simulator, aggregator and strategy are implemented as single-threaded microservices – these are extremely simple and do not depend on Chronicle Queue or Chronicle Wire. The inputs and outputs of each microservice are defined as Java interfaces – each microservice implements its input interface and composes its output interface, and these interfaces in turn refer to Java DTOs which are sent between the microservices.

Because each hop is fully recorded, trading firms can later load the persisted queues into an analysis tool, correlate them with an external hft dataset, and precisely measure latency to verify compliance with quality measures demanded by regulators.

For the aggregator service the interfaces look like this:

And

In this simple example there is only one method on each, but a service can implement many interfaces with many methods and any number of arguments of all kinds, including primitives.

One of the DTOs is:

And the microservice:

You can see that there is no dependence on any Chronicle code, with the exception of the DTO, and the microservice respects the Unix philosophy – it is simple and easy to understand, and does Just One Thing. Chronicle Wire Method Readers take care of reading incoming events from the “in” queue and dispatching these to the “in” interface methods, and Chronicle Wire Method Writers ensure that when the service calls a method on the “out” interface, the method call is serialised and written to the out queue.

The end-to-end path therefore stays deterministic even under bursts of latency activity typical of market events millisecond by millisecond.

All input and output to/from the service is serialised to/from Chronicle Queues using BinaryWire which is compact, efficient, fast and zero-garbage and yet still self-describing, so its data can be read by any other microservice or tool – you can see this by trying to run the examples and the “tailf” command below.

In live trading this property lets agency algorithms, proprietary trading engines and even external analytics modules subscribe to the same market data stream without jeopardising throughput or adding measurable latency.

All this code can be found in the Chronicle-Queue-Demo/md-pipeline module.

The DTO extends a Chronicle Wire class SelfDescribingMarshallable, which provides functionality including:

• Automatic serialisation using Chronicle’s Wire and Bytes Marshallable strategies

• Automatic conversion to/from friendly YAML format using Chronicle Wire

• Transparent support for versioning – new fields can be added to a DTO and old fields removed without causing breakages, and code can be plugged in to convert from an older version to a new version

• Support for renderers e.g. transactTime is a microsecond timestamp stored efficiently as a long but rendered to the user by the MicroTimestampLongConverter to a friendly date/time string

• A path from greatest convenience (Chronicle Wire) to lowest possible latency (Bytes Marshallable) – our recommendation is to start with the SelfDescribingMarshallable’s implementation of Wire serialisation and you can incrementally convert DTOs in the fast path to use Bytes Marshallable

This layered approach empowers developers to fine-tune performance as their strategies evolve, ensuring the system can scale with frequency trading volumes while preserving low jitter and consistent, predictable latency.

Testing

The microservice leverages the above functionality, and Chronicle Wire’s YamlTester to allow very simple behaviour-driven YAML testing – the test class looks like this:

And the “aggregator” folder contains an in.yaml file

Which the YamlTester plays into the AggregatorImpl class, automatically deserialising the MarketDataIncrement DTOs and dispatching them to the mdi method. This is all done in a single thread, allowing breakpoints to be set. The YamlTester records any output sent to the “out” interface and compares it to the contents of the out.yaml file. Intellij shows a friendly text diff in case of failure, making it very easy to see what was changed:

This methodology not only validates business logic, it also allows teams to measure latency with microsecond precision, ensuring that code changes do not introduce increased latency activity that could harm market quality or disrupt sensitive trading strategies.

Hooking it all up and running

The sample code contains maven exec java stanzas to run each microservice and also stanzas to run the ChronicleReaderMain tool which reads messages from the queues, deserialises them and displays them as YAML. To run:

Start up three terminal screens and run the following in the md-pipeline directory to start the services

And to watch the output from each service start up three more screens (these are the equivalent of Unix tee and tail -f)

Using this approach, developers can observe how strategies respond market changes in real time, verify the integrity of depth limit book construction, and calculate market quality measures such as spread, liquidity and order-to-trade ratios.

Conclusion

We can see that it is possible to realise the Unix Philosophy in Enterprise IT using a strongly-typed Enterprise language (Java), a suitable component technology (microservices) and an appropriate mechanism to glue them together (Chronicle Queue & Wire).

For frequency trading and latency trading applications, these principles translate directly into systems that achieve sub-microsecond round-trip times, uphold robust market quality, and maintain deterministic behaviour even during extreme market events. Chronicle’s approach empowers proprietary trading desks, agency algorithms and other market participants to build strategies that can adapt to evolving market structure, exploit short term volatility and optimise trading frequency while safeguarding against the adverse impact latency activity can have on liquidity and price discovery.

Note that if you want more features you can talk to [email protected] about commercial extensions – Chronicle Services – which provide the following features:

• HA & DR

• Sophisticated restart and replay strategies

• IoC runtime

• Health monitoring and latency stats that help measure latency in production and drive continuous improvement of market quality measures

• Services visualisation

• Configuration

• Timers

—

¹ Actually any language that runs on the JVM. There is also a C++ version of Chronicle Queue that also supports Python, nodeJS etc.