Navigating Behaviour with Events
December 5th, 2023
Two Approaches
There’s little doubt that modern software architectures lean towards asynchronous, event driven programming models for communication between distributed components, where real time responsiveness is increasingly expected. Such architecture event driven patterns are vital when components—whether microservices, monoliths or serverless functions—are loosely coupled and may reside on different machines or across different cloud regions. Synchronous models such as Remote Procedure Call (RPC) are now largely discredited [Waldo, Vinoski]; even asynchronous variants of these are not seen as desirable, primarily because they impose a request response mindset that limits scalability, fault tolerance and real time processing.
When we strip away the layers that present the illusion that RPC is really just the same as Local Procedure Call (i.e. a normal method invocation), we find, sitting above the transport, some form of messaging protocol—often an event bus or event broker implementing a publish subscribe model. And these days, the APIs seem more willing to expose developers to these lower-level details, making it clear to programmers that communication with a remote component is inherently more complex than making a simple procedure call and that data consistency must be handled deliberately in any event driven systems.
Event-driven approaches refine and extend these ideas of basic messaging interactions by introducing more semantic significance to the actual messages that are being sent between components, and also by providing support for persisting messages in perpetuity through event channels or event stream logs. An event can be posted with minimal overhead to some transport mechanism—often a high-performance data streaming platform such as Apache Kafka or Chronicle Queue—and received by one or more event consumers from this transport, again with minimal overhead, paving the way for true real time event propagation.
Semantically, an event is an indication that something has happened. This could be at the business level, or at the infrastructure level. Either way, an event is intended to be an immutable event source record of the fact that something has happened, and in a fully Event-Driven Architecture (EDA) approach, this “fact” will never be lost. Such durability underpins data streaming strategies and makes later event stream processing or even complex event processing straightforward.
There are numerous advantages to following an Event-Driven Architecture:
The sender and receiver are decoupled in both time and space – an event can be posted even if no listeners are currently active, and without any knowledge of the physical location of the receiver, delivering natural loose coupling for all components event flows.
Minimal overhead in posting and receiving events supports higher rates of throughput with better vertical scalability, real time processing and responsiveness, key advantages event driven practitioners value.
Persistent event streaming stores support replaying scenarios to aid in debugging, simple event processing and complex event reconstruction alike.
Diagnostics are easily available from the event stores, which double as a centralised event broker and audit log.
Components’ state is represented as a deterministic sequence of updates rather than simply the most recent values, providing audit trails, ensuring data consistency, and making restarting following failover much more straightforward—a cornerstone of fault tolerance in driven architecture event systems.
Testing can be performed using data captured from real scenarios, enabling real time data replays that highlight regressions quickly.
While the Event Driven Architecture approach provides an effective means of implementing systems, it also works well with the Behaviour Driven Development approach of designing the business logic of an application, often complementing each other’s strengths and weaknesses. By marrying EDA with BDD you gain not only resilient runtime behaviour but also a living specification that evolves at the pace of your event driven applications.
Behaviour-Driven Development provides a way of capturing requirements and constructing associated test cases using language that is associated with the problem domain, and is therefore more likely to be accurate from a customer perspective. This focus on a common language leads to an approach for describing tests related to user scenarios, following the following pattern, which naturally aligns with command query responsibility ideas and the broader query responsibility segregation mindset:
Given: the state of the component(s) being tested
When: the user applies some action
Then: the state of the components should change to this
In the Event-Driven Architecture, it is usual to manage the state of a component using an approach known as Event Sourcing. Since state changes are triggered by and notified to other components using events, we have the ability to (re)construct the state of the component at any point in time, simply by replaying the state-changing events from some known point in time (either from the start of the application or from some state snapshot) to the required time. This technique underpins event stream processing pipelines and empowers downstream event consumers to build their own projections in near real time.
So the BDD test pattern described above can be described using events:
Given: the state constructed by applying a sequence of events
When: a further event is applied to the component(s)
Then: the component should post an event indicating the state change
The events used in the test case can be defined and stored in a text format, for example in files. The testing framework may then use these to run the test(s) on the component in a deterministic, data-driven fashion—very much in line with simple event processing best practices.
The scenarios may relate to a single component, a subset of components in the application or to the application as a whole, and present a number of advantages:
A clearer focus on business requirements, with attendant reduction in ambiguities, improving overall architecture EDA software quality.
Rapid feedback loops with early detection of issues, which is critical for real time trading or stream processing systems.
Continuous integration testing that validates every new branch of functionality without compromising data consistency.
While these two approaches do not mandate each other, they do tend to work well together, and the Chronicle Services framework provides developers with support in building components and applications that follow the guidelines.
Events, Events, Events
Some event driven architectures draw distinctions between certain broad types of messages/events sent between components, for example, CQRS (Command Query Responsibility Segregation). It is perfectly reasonable to consider that all communications between components are events, even given the idea of an event as being an indication that something “has happened”:
A change has occurred in the state of a component – a classic event source update.
A “Command” to perform some action was received, illustrating command query responsibility.
A “Query” was received, representing the read side of query responsibility segregation.
It is even possible to implement request response-type communication using events, by using appropriately typed events that are generated and handled with regard to retry, correlation logic and error handling. Implementers can layer complex event processing or stream processing stages on top of these basic interactions to derive higher-level insights.
Reference data and component configuration can be established using events. Normally we would expect such events to be handled before handling other types of events during a component startup (or restart). However, they could also occur during operation, meaning that they would be persisted along with other events, allowing analysis of changes in the behaviour of components based on configuration changes in a reproducible way. This creates a single, authoritative ledger of configuration and runtime facts—an invaluable capability for governance and root-cause analysis within large-scale event driven systems.
The passage of time is something that is normally important to components. It is also something that can be represented through input events to these components. The source of these input events may be a “real” wall clock, or some other means of representing time moving forward that can be substituted during testing and debugging, maintaining determinism during replays of events. This event-based notion of time proves particularly useful in real time trading platforms where deterministic latency and accurate sequencing of real time data are paramount.
Logging functionality is a natural fit for implementation using events. Indeed some of the most widely used event streaming frameworks grew from a requirement to consolidate and manage logs in large distributed applications. Conventional approaches to logging can make it difficult to reproduce issues due to the fact that the log messages occur “out of band” in relation to the component flows, and can affect outcomes. Handling logging of errors through the main event processing infrastructure makes these deterministic and simplifies error handling and post-mortem analysis.
There is also a larger discussion to be had around logging in general, and its frequent overuse to display informational messages, which can pollute output that could still contain messages necessary to identify and help solve error situations. The ability to obtain such information from the event management infrastructure can help reduce this extraneous output, making it easier to see when a situation has occurred that needs attention and highlighting the advantages event driven platforms provide for operational observability.
Of course, not every component is necessarily going to follow these guidelines. Some form of interaction with components—for example, external systems that are not event driven—will be likely. This could be talking to a relational database, an external REST API, or to a UI. Gateway components may be used to manage this interaction, mapping between event-based communication and an alternative, perhaps request response model, without losing any information in our event store. These gateways effectively act as event producers on one side and event consumers on the other, preserving loose coupling.
Putting it into Practice
At Chronicle, we have many years of experience in building low-latency Java applications using our world-beating libraries. Distilled from this experience, we have developed Chronicle Services, a framework that supports the development of high performance Java microservices, following the guidelines of event driven architecture and Behaviour Driven Development, to build applications that perform with world beating latency performance and sub-millisecond real time guarantees.
A combination of externalised configuration, powerful APIs and flexible deployment options provides access to enterprise-class features such as high availability, fault tolerance, data consistency, observability and elastic scalability. Combined with a novel approach to testing based on BDD techniques, this allows developers to concentrate on implementing business logic while the framework takes care of stream processing intricacies and error handling. Development timelines are shortened without sacrificing quality and ROI is apparent much more quickly.
With Chronicle Services, the primary unit of computation is the Service, which receives inputs in the form of events from zero or more sources—these may be internal event producers, external systems pushing real time data, or event channels bridged from technologies like Apache Kafka—and posts output, again in the form of events, to a single sink. The default transport for events with Chronicle Services is Chronicle Queue – a persisted, shared memory channel for inter-process communication offering class-leading latency performance of under 10 microseconds from write to read at the 99.9 percentile, a hallmark of efficient data streaming.
Events are represented as POJOs and are persisted to a Queue using a highly efficient proprietary format, with marshalling and unmarshalling being completely transparent. Chronicle Services does not distinguish between different “types” of events, for example CQRS as described above, preferring instead to consider everything as an event. This simplifies the processing requirements of event management, reinforces loosely coupled design and produces a single, append-only log suitable for downstream event stream analytics.
Chronicle Services also includes a BDD-based testing framework, known as YAMLTester. Tests are written following the guidelines described above, with separate YAML files containing the events used to specify the three components of each test:
Given: setup.yaml
When: in.yaml
Then: out.yaml
A test is run using the JUnit testing framework, with helper classes that:
Create an instance of the service, initialising state using the events in the setup.yaml file—effectively replaying the event source history.
Pass the events in the in.yaml file to the service instance, simulating simple event or complex event inputs as appropriate.
Capture the output events and compare them with the expected output specified in the out.yaml file, ensuring behavioural alignment and automatic error handling checks.
Results can be displayed in a terminal, or within an IDE such as IntelliJ. Failing tests are reported with additional details of mismatches between expected and actual output, ensuring advantages event driven methodologies bring to real time quality assurance.
Having input and output events kept separately makes it easier to keep track of output for variations of invalid input, which in turn makes regression testing easier and supports continuous event streaming validation pipelines.
Summary
The practices of Event-Driven Architecture, with its focus on loose coupling, publish subscribe patterns and event streaming, and Behaviour-Driven Development have more in common than may at first be apparent. At Chronicle, we have embraced both of these approaches and implemented them through our Chronicle Services Framework, crafting EDA software that powers some of the most demanding real time processing environments in global finance.
Experience, both internally within Chronicle and of our customers, has shown that our approach is beneficial in both improving the quality of the software being developed and reducing development timescales. By leveraging architecture EDA software principles, embracing event stream processing and automating error handling and testing, teams can innovate faster, maintain stronger data consistency, and realise the full advantages event driven solutions can deliver.