Software Continuity: Failover Strategies for Uninterrupted Operations

In today’s ever-changing and fast-paced digital landscape, maintaining uninterrupted business continuity is of utmost importance for market participants across the financial industry. For capital markets firms, trading venue participants and other financial industry participants, a well-tested business continuity plan that aligns with regulatory requirements and sound practices in risk management, cyber security, and incident management is now a baseline expectation. Enter Chronicle Services, a robust mechanism dedicated to upholding application integrity even in the face of service disruptions. With Chronicle’s technology at the core of many critical systems, financial services organisations can establish continuity plans that keep trading platforms and existing systems running in real time, even during a major operational disruption. Let’s examine the world of failover strategies and explore how they safeguard software continuity, firm business continuity, and overall business operations.

Stateless vs. Stateful Services

It’s important to appreciate the difference between stateless and stateful services, especially when building continuity planning software or integrating with existing continuity management tooling. Your choice of failover approach is highly influenced by this understanding, particularly if the aim is to resume services seamlessly from where they were interrupted while preserving compliance with policies procedures that govern business continuity planning within financial institutions.

A stateless service, which can be likened to a pure function in functional programming, handles requests independently of previous occurrences and maintains no mutable state between them. The service will always produce the same output when supplied with the same input value(s). While the appeal of simplicity and lack of ordering concerns are enticing, not all facets of an application can be represented as stateless, hence the requirement for stateful services that underpin electronic trading, post trade processing, and other systems trading workflows.

Contrarily, output from a stateful service depends on information from past invocations. Echoing the notion of side effects in functional programming, the output of a stateful service may differ for identical input due to this mutable state. In the event that a service fails and requires to be restarted, it is necessary to implement a strategy to recreate the service’s state as it was when it was stopped, thereby supporting disaster recovery objectives that regulators require trading venues to meet.

Event stores such as Chronicle Queue offer powerful and efficient ways of achieving this. All mutations to a service’s state are posted as events, and it is straightforward to reconstruct the state by replaying these events. The fast and efficient event processing of Chronicle Queue makes the task of replaying events manageable, even when volumes soar during periods of high-volatility trading or crisis management scenarios.

However, complexities might crop up when alterations are made to your business logic across different versions of your services. In such situations, the same input might yield a different output, creating possible problems in maintaining uniform system behaviour. It’s crucial to remember here that while events that update the state are deterministic, business logic, which generates the events, might not be. Therefore, while the outcomes (events) of what you did (business logic) stay the same, the reasons why you do things (business logic) could evolve. Recognising and documenting these nuances in a continuity plan supports both risk management and business continuity management processes.

One effective mitigation strategy involves periodic snapshotting of system state into the output queue. By integrating these snapshots into the event queue, you establish consistent recovery points that capture the state of your system at specific intervals and align with the Recovery Point Objective (RPO) criteria that most business continuity plans in banking capital markets mandate.

If a system restart or recovery occurs, the state can be rebuilt from the latest snapshot instead of replaying the entire history of events. This approach reduces the amount of data that needs to be processed, speeds up the system recovery time, and shortens the potential downtime that can impact trading venue systems and their venue participants.

Still, it’s crucial to strike a careful balance with the frequency of snapshots. This balance depends on factors such as data volume, business logic throughput, and desired recovery time, while also considering the storage costs and resources needed to create and manage these snapshots. Note: Snapshot generation can be time consuming, so Chronicle Services can support generation of snapshots on a background thread, ensuring that operations continue in real time without introducing latency that could compromise critical systems.

In summary, an amalgamated approach of efficient event replay using a tool like Chronicle Queue and strategic snapshotting can offer a robust solution for managing state in a continually evolving event-driven architecture. Note: It is also possible for Chronicle Services to handle replaying from the output queue and not just the input queue. These combined techniques form the backbone of many continuity disaster recovery strategies used by leading trading venues and financial trading organisations.

The Challenge of Failover: Rebooting Services and Reestablishing State

In failover situations, the critical task is rebooting a service and restoring its local state to align with the prior service instance. Chronicle Services provides tools for successful failover, specifically focusing on restoring state in stateful services. These capabilities enable financial institutions to satisfy both internal and external regulatory requirements that emphasise resilience, continuity planning, and the protection of market integrity.

Reinitializing a service and restoring its local state to reflect the preceding service instance’s state are key challenges in failover situations. This is particularly pertinent for stateful services, where consistency is a necessity for risk-sensitive electronic trading and post trade workflows that underpin capital markets.

The complexity is heightened when a warm standby instance is involved in the failover. The standby instance, kept in sync with the primary service, is poised to take over immediately. The challenge involves synchronizing the primary and standby instances and facilitating the standby’s seamless transition to primary mode during failover. This process demands meticulous coordination to ensure service continuity, which includes traffic rerouting and final state changes synchronization after the last update to the standby. Such alignment is integral to firm business continuity and to maintaining confidence among market participants during any major operational disruption.

Consider an application with three services: Service-a, Service-b, and Service-c. While Service-a and Service-b are stateless, Service-c is stateful with a high sensitivity to event processing order. Ensuring each of these services follows documented continuity plans helps organisations demonstrate adherence to business continuity planning guidance issued for trading venues and their intermediaries.

In the wake of a service restart following a failover, it’s crucial to restore its internal state to match the point of failure. Chronicle Queue‘s dual role as an event transport and persistent storage mechanism enables this by replaying events stored on disk, whether those files reside within a local data center or a geographically diverse disaster recovery site.

However, the reliability of this process depends on successfully syncing events to the disk before the primary service fails. In rare situations, such as kernel panic, this sync might not be assured. Unwritten data at the time of system failure might be lost, affecting precise state reconstruction during recovery. Implementing safeguards for regular and reliable data synchronization from memory to disk can minimise the risk of sudden data loss. These safeguards—often outlined within policies procedures for continuity planning—are essential to protecting trading systems that must operate in real time across global markets.

An effective risk mitigation strategy involves keeping a safe copy of the data on a separate machine, acting as a ‘warm standby’. This machine can be within the same network or geographically separate, to protect against simultaneous failures. Having a near up-to-date state of the system at failover time, the standby machine enables quicker recovery. To handle the case of the primary going down while messages are in-flight, consumers of the queue can be set up so they do not see the message in the queue until a safe copy of the message has been successfully acknowledged from the secondary. Adopting this robust approach ensures that no messages are ever lost, but it comes at the cost of accepting the round-trip network latency—a trade-off many financial services firms willingly embrace to meet strict continuity disaster recovery objectives.

The restart of a service after a failover necessitates the alignment of its internal state with the point of failure.

The ‘History Record,’ an integral part of event replay, includes meta-information about each event, such as the originating service and posting time. Configuring the ‘startFromStrategy’ and ‘inputsReplayStrategy’ fields allows control over event processing order and choice of events for state reconstruction. Such granular replay logic is invaluable for business continuity management teams who need to demonstrate that critical data is handled in accordance with continuity management frameworks.

However, maintaining exact event order can be complex with multiple input queues. To navigate this challenge, Chronicle Services uses History Records for each input event, ensuring a strict time-ordered replay regardless of the queue origin. This discipline helps satisfy auditors that continuity planning software operates with deterministic precision—a key factor in preventing business continuity disaster scenarios in capital markets.

Addressing Stateless Services in Failover Scenarios

In contrast, implementing failover for stateless services like Service-a and Service-b is relatively straightforward due to their lack of internal state. Upon restart, these services can directly read new events from the queue, conveniently ignoring those that don’t impact their state. For electronic trading gateways or API adapters that fall under this category, the simple design reduces crisis management overhead and simplifies disaster recovery drills for venue participants.

This default behaviour, characterised by the ‘startFromStrategy’ property’s default value of ‘LAST_WRITTEN,’ obviates the need for event replay in stateless services. It also aligns with continuity planning principles that emphasise reducing complexity in low-risk service tiers to streamline firm business continuity efforts.

In conclusion, by harnessing the power of Chronicle Services, both stateful and stateless services can seamlessly recover from failovers, ensuring robustness and high availability. The thoughtful interplay between service design, event replay, and strategic event handling mechanisms guarantees that the application state is safeguarded and accurately replicated during disruptions. As a result, continuity of service and operational efficiency are promoted, providing a resilient foundation for your software ecosystem—one that aligns with the stringent business continuity plans, regulatory requirements, and sound risk management practices demanded across trading venues, banking capital markets, and the broader financial services sector.

Resources

For further details and examples of Chronicle Services, see the documentation portal here.