Distributed Matching Engine Instances, & Aggregating Services

Matching Engine instances can be freely scaled to meet design load: the only requirement is that all transactions for any one symbol are handled by a single instance. The Transaction Queue feeds from multiple instances can be used to feed other services, and these services can either themselves be distributed, or aggregated into a single instance. Where aggregation is required, Transaction Queues from multiple instances are replicated into the aggregating service, then polled for updates.

Risk Controls

Risk checks can be handled either sharded or aggregated. This is illustrated in the below diagram where three transaction engines replicate their transaction queue to the Aggregating Risk service host. The Risk service then uses those queues to maintain a global risk state which is persisted by a series of atomic updates to an Aggregated Risk Queue (directly analogous to how Transaction Queues are built by the Chronicle Matching Engine instances). This Aggregated Risk Queue is then replicated to each of the instance’s hosts, where it is polled by the instance event loop for updates.

A number of risk control types are supported - details have been omitted from this document.

Drop-Copy Service

A Drop-Copy feed can be built in a directly similar fashion to an Aggregating Risk service, although in this instance there is no need to replicate back to the Chronicle Matching Engine instances: any output from the Drop-Copy Service is sent directly to the connected clients.

Consolidated Book

A Consolidated Book Service providing a single view across all Books (for example for driving front-end GUIs) can be assembled following exactly the same approach as a Drop Copy service. In this case the Consolidated Book service monitors (replicated copies of) the Chronicle Matching Engine instances’ queues and assembles a real-time summary picture across all books, which can be used to drive queries and live updates to connected clients. Again optionally this service can share the same input queues as Risk and/or Drop Copy.

Market Data Service

A Market Data service can be constructed in a similar fashion to the above as an aggregating service using replicated copies of each Matching Engine’s output. Alternatively, each Matching Engine host can publish market data individually given all information is available from the local Transaction Queue for the subset of symbols traded on that instance. Market Data updates would normally be published via UDP. A separate TCP/IP interface is provided to enable recovery of messages following a gap detection on the UDP stream. This mechanism will provide the ability to recover messages only up to a limited age (e.g. last N messages, or last hour, or trading day), after which the message should be considered lost.