A Load Balancer, FIX Version Translator, and Rule-based Drop Copy
FIX (Financial Information eXchange) protocol was developed to provide a reliable and high-speed exchange of transaction information between trading parties. In a typical scenario, there are point-to-point connections between clients and servers, ie a firm may run one or more FIX servers and multiple counterparties (clients) connect to them. As the generated traffic by each client varies, this topology might result in unbalanced distribution of load on the server side and consequently performance degradation. Figure 1 shows a typical scenario of FIX client and server connections.
Figure 1 – Connections between FIX clients and servers in a typical scenario
A further potential issue with the point-to-point topology is that the FIX protocol is an open standard and has been evolved continuously so several versions of the FIX protocol exist and may be developed in the future as well. This topology requires both server and client to use the same version to communicate. Further complications might emerge during version upgrades considering that client and server generally will not be controlled by the same authorities.
As described in the next sections, Chronicle FIX offers router functionality that can address both issues and provide additional features. In summary, the FIX Router is a multifunctional application with the following benefits:
- Load balancing to help distribute load across multiple internal servers transparently to clients
- Allows a site to present a constant connection surface to clients, hiding the internal layout of machines
- De-risk internal server updates (eg by routing only a subset of flow onto new/updated servers)
- Provides a rule-based Drop-Copy service
- FIX version translator/adaptor
The FIX Router Functionalities
The FIX Router can route messages to destinations based on standard and bespoke tags. Routing rules are configurable and are defined based on the values of one or more tags. Upon receiving a message, tags of the message are compared to the routing rules. The first matched rule is applied to forward the message to a destination defined by the rule. Figure 2 shows the same components in Figure 1 communicating through a FIX Router.
Figure 2 – A FIX Router routes FIX messages between clients and servers
A sample routing rule defined in a YAML configuration file is indicated below which specifies that if tag 49 and tag 21 are set to “Server1” and “1” respectively the message is sent to Server 1.
rule1: { pattern: '$49=="Server1" and $21=="1"' policy: ORDERED, routes: [ "Server1" ] },
Optionally, tag numbers can be associated with arbitrary names so the names can be used to specify routing rules instead of tag numbers for making the rules easily comprehensible.
The rule definition follows a natural syntax; it supports boolean operators (ie ==, !=, >, <, >=, <= ) and regular expressions. In addition “and”, “or”, “not” for forming compound logic expressions as well as “has” to test the presence of a tag are supported.
Chronicle software provides ROUND_ROBIN and ORDERED routing policies however this property is extensible. ORDERED policy always selects the first available route specified in the list of routes for all incoming messages. ROUND_ROBIN selects routes in turn with each incoming message.
It is worth noting that the routing is “sticky” based on ClOrdID ie, all activities (amend, cancel, etc) on the same NewOrderSingle routes to the same destination as the original NewOrderSingle.
Load Balancing
To configure a router as a load balancer in the above topology, a routing rule can be defined as follows so that every incoming new order is sent alternately to Server 1 or Server 2 using a round-robin policy.
rule2: { pattern: "true", policy: ROUND_ROBIN, routes: [ "Server1", "Server2" ] },
Version Translation
A FIX Router can be configured to automatically translate FIX messages of different versions between sessions. Therefore, if client and server engines work with different versions of FIX, an intermediate router can translate messages to the appropriate version. For example, in Figure 2, it is not necessary for clients and servers to work with the same FIX version to be able to exchange messages. To make them interoperable, the router must be configured with the appropriate FIX version for its sessions and the client and server FIX engines do not need to be altered.
Rule-based Drop Copy
Logging FIX messages is important for later tests and support purposes also for production back-office, reconciliation feeds, etc. However, not every message might be useful for an inspection so it is desirable to log only messages of interest. In addition, it is beneficial to log messages in different log files according to specified criteria eg, logging all newOrderSingle in one log file.
The FIX Router can be configured to act as a rule-based drop copy tool and logs its outbound messages (messages from router to servers and clients) selectively in persistent queues. The rules to select a message to log are specified based on expressions in the message tags similar to the routing rules expressions as explained above. A router supports logging messages in several logs simultaneously while each one logs messages that match a specific expression.
The functionalities of Chronicle’s FIX Router are not limited to the above categories; a noteworthy use case in practice could be where the Router is used to manage and de-risk production server upgrades. For example, a subset of flow (as determined by matching rules) can be steered onto upgraded server instances, with the rest on the old version.
An instance of the FIX Router can supply all the above functionalities simultaneously.
Performance
This section presents the results of performance tests on the FIX Router as well as tests when a client and server engine communicate directly with no router in between for comparison.
The performance of a FIX Router was evaluated in a scenario where a client sends NewOrderSingle messages to a router and the router routes the messages to a server that replies with ExecutionReports. The results were produced on a Ryzen 9 3.4GHz machine with core pinning and CPU isolation which are shown in Figure 3 where:
- end-to-end: client--> router –-> server –-> router –-> client
- client2server: client –-> router –-> server
- server2client: server –-> router –-> client
Figure 3 also shows the results of the same test when the client and server directly communicate with no router in between (columns with pattern fill). Comparing the results with/without the router suggests just a ~12 microseconds increase in latency for a message crossing the router (pink and purple columns). This 12 microseconds delay includes the delay created by the network layer too, not just the FIX Router application. The graph also indicates at most 8 microseconds increase in the end-to-end latency from 50 to 99.9 percentile.
Figure 3 – Message latency in scenarios with and without the FIX Router
Conclusion
This article introduced Chronicle’s FIX Router and some of its functionalities. The FIX Router provides a number of important capabilities in managing a distributed FIX environment, and supports highly configurable rule sets. The results of performance tests indicate the Router adds only around 12 microseconds overhead to FIX message delivery when each engine (client, server and the Router) is hosted on a separate machine. Latencies when routing to servers on the same host as the Router (where host-host network overhead can be eliminated) will be significantly less.
References
