Data Driven Testing of Chronicle Services

Introduction

In earlier articles we have seen how Chronicle Services is a framework supporting the development of applications based on Event-driven microservices. Individual services operate by processing events read from one or more input queues, and posting events reflecting the result of their processing to their output queue.

Clearly, an important aspect of developing services is the ability to test that they are functioning correctly. A service operates as a “black-box” – interaction with other components is based solely on its external interface, which describes the events/messages it can accept and those that it will post. Testing the service can be done along these lines, using some built-in helper functionality provided by Chronicle Services.

Chronicle Services’ testing support follows the guidelines of Behaviour Driven Development (BDD). We have found that this approach combines well with Event-driven architectures, as discussed in more detail here.

In this article we will examine the Chronicle Services testing framework in more detail.

The Chronicle Services Test Framework

Chronicle Services’ testing framework focuses primarily on functional testing of individual services, although it can be adapted to perform integration testing as well. As mentioned above, the approach taken is akin to “Black Box” testing, where we verify that for a given input event, the service produces a specific output event or events. 

Using Chronicle libraries we can specify these events using YAML, and a framework called YAMLTester will

• create an instance of the service under test
• initialise any require state to process the input event
• invoke the handler for the input event
• capture the output event and compare it with the expected event

Testing is performed using familiar components from the JUnit framework. Tests are data driven, designed according to the guidelines of Behaviour Driven Development where each scenario is described using the template

Given:  the state of the component(s) being tested

When:  the user applies some action

Then:   the state of the components should change to this

YAMLTester allows us to define each of the above steps using YAML files, which describe events that are either input to the service or expected to be output from the service:

Given: (the state of the component(s) being tested) setup.yaml

When: (the user applies some action) in.yaml

Then: (the state of the components should change to this) out.yaml

An Example

As an example, let’s look at the service discussed in a previous article. The service under test manages account balances based on input transactions to either credit or debit the account. We’ll focus on the specific aspect of the service that applies a transaction to an account under management. We’re not looking at error situations for the moment, these will be covered in a later article. The functionality can be represented in the following diagram:

From a BDD perspective, this scenario can be written as:

Given: an account named “acc1” with number 1234 exist with a balance of 100

When: a CREDIT transaction with value 100 is applied to this account

Then: an event is posted reflecting an updated balance of 200

Since the Transaction and Account DTOs are defined as classes that extend AbstractBaseEvent<E>, we can see that Chronicle Services attaches a timestamp to the event being transmitted. When running in the Chronicle Services runtime, this timestamp is guaranteed to be unique and is used as a means of attaching a unique ID to every event as it is posted. When running in the test harness, where time is not critical in the correctness of the service, we can include a “stub” timestamp, as shown below.

Data Files for the Test

The Given clause for the test is represented in the file setup.yaml, for the test case shown above we need to ensure there is an account with number 1234, and a balance of 100.0:

---
accounts: [
 !Account {
   eventTime: 0,
   name: "currentAccount",
   accountNumber: 1234,
   balance: 100.0
 },
]
...

The When clause is represented in the file in.yaml, and specifies a transaction that will credit 100.0 to the account with number 1234:

---
transaction: {
 eventTime: 0,
 accountNumber: 1234,
 entry: CREDIT,
 amount: 100
}
...

The Then clause, representing the expected output for the test case, is defined in the file out.yaml. It should show the account with number 1234 now has a balance of 200.0:

---
onTransaction: {
 eventTime: 2023-11-24T16:56:41.345678,
 name: currentAccount,
 accountNumber: 1234,
 balance: 200.0
}
...

Notice that the events shown all carry a property called eventTime, however the test framework focuses on the other properties in each of the events. We are not testing time related functionality here so the values are not checked.

Adding Comments

Yaml files may contain comments, introduced using the # character. If a comment is added to the input file on the line preceding a test, then YamlTester will copy the comment line unchanged to the output, and display them ahead of the appropriate tests’s output. To avoid failures, the expected output file should contain these comments exactly as they are in the input file.

For example, if the input file were to contain two input events commented individually:

# CREDIT transaction should add amount to balance
---
transaction: {
 eventTime: 0,
 accountNumber: 1234,
 entry: CREDIT,
 amount: 100
}
...
# DEBIT transaction should subtract amount from balance
---
transaction: {
 eventTime: 0,
 accountNumber: 1234,
 entry: DEBIT,
 amount: 50
}
...

Then the expected output file needs to include the comments as well as the output events:

# CREDIT transaction should add amount to balance
---
onTransaction: {
 eventTime: 2023-11-24T16:56:41.345678,
 name: currentAccount,
 accountNumber: 1234,
 balance: 200.0
}
...
# DEBIT transaction should subtract amount from balance
---
onTransaction: {
 eventTime: 2023-11-24T16:56:41.345678,
 name: currentAccount,
 accountNumber: 1234,
 balance: 150.0
}
...

Running the Test

The test is run using the JUnit 4 framework. Therefore we need to supply some scaffolding to initiate the test:

public class BalanceTest {
 private static void testMessages(String directory) {
   try {
     SystemTimeProvider.CLOCK = new SetTimeProvider("2023-11-24T16:56:41.345678");
     YamlTester.testMessages(directory, TransactionSvcOut.class, TransactionSvcImpl::new);
   } finally {
     SystemTimeProvider.CLOCK = SystemTimeProvider.INSTANCE;
   }
 }
 @Test
 public void testBalance() {
   testMessages("test/balance");
 }
}

The test is annotated using JUnit’s @Test annotation. It invokes the local method testMessages(), which accepts an argument specifying the directory where the data files can be found. The main work is performed by the YamlTester.testMessages() method, which accepts three arguments:

• The directory where the data files are located
• The output API for the service under test
• An expression that will create an instance of the service under test

An instance of the service is created, and the events specified in the setup and input files are posted to its input queue. Output events are captured, output DTOs are created and these are compared with those specified in the “expected output” file. If there are any discrepancies the test is marked as failing.

The test can be run from within an IDE, if successful the test frame will display any output:

If there is a failure, for example if the generated balance did not match the expected output, then a failure indication will be shown:

By clicking on the link in the output, a comparison window will be shown, which allows this failure to be investigated:

The test can also be run from the command line, using maven. 

In  the success case:

$ mvn test
…
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running software.chronicle.services.cookbook.example2.BalanceTest
[main] INFO Jvm - Chronicle core loaded from file:/Users/george/.m2/repository/software/chronicle/chronicle-services-all/2.25ea0/chronicle-services-all-2.25ea0.jar
[main] INFO TransactionSvcImpl - Adding account 1234 with initial balance: 100.0
[main] INFO TransactionSvcImpl - Applying CREDIT of 100.0 to account 1234
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.042 sec
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
…

And in the failure case:

$ mvn test
…
-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running software.chronicle.services.cookbook.example2.BalanceTest
[main] INFO Jvm - Chronicle core loaded from file:/Users/george/.m2/repository/software/chronicle/chronicle-services-all/2.25ea0/chronicle-services-all-2.25ea0.jar
[main] INFO TransactionSvcImpl - Adding account 1234 with initial balance: 100.0
[main] INFO TransactionSvcImpl - Applying CREDIT of 100.0 to account 1234
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.028 sec <<< FAILURE!
testBalance(software.chronicle.services.cookbook.example2.BalanceTest)  Time elapsed: 0.978 sec  <<< FAILURE!
org.junit.ComparisonFailure: expected:<...r: 1234,
  balance: [3]00.0
}
...> but was:<...r: 1234,
  balance: [2]00.0
}
...>
at org.junit.Assert.assertEquals(Assert.java:117)
at org.junit.Assert.assertEquals(Assert.java:146)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
…

Once again we can see details of the parts of the actual output that differed from the expected.

Working with Time

Chronicle Services Events contain a payload defined in a DTO. The DTO should be defined to extend the AbstractBaseEvent<E> class, which adds a timestamp property to each event. This timestamp is added by the Chronicle Services runtime when an instance of the event is posted. It is not generally required for the purposes of testing, since this should only examine the functionality of the service related to the event payload. Nevertheless if the expected output does not include a timestamp then the test will be considered to have failed as the actual event payload will not match the expected.

Chronicle Services does, however, use a component called a TimeProvider to generate timestamps rather than directly accessing the system’s time APIs. The TimeProvider is a layer of abstraction over time functionality, which by default passes requests through to normal system level functionality. However we can replace this behaviour with a different TimeProvider that returns timestamps of the correct types, but allows us to specify what the time value will be. This makes it easy to test expected output events that contain timestamps.

Recall the test scaffolding code:

 private static void testMessages(String directory) {
   try {
     SystemTimeProvider.CLOCK = new SetTimeProvider("2023-11-24T16:56:41.345678");
     YamlTester.testMessages(directory, TransactionSvcOut.class, TransactionSvcImpl::new);
   } finally {
     SystemTimeProvider.CLOCK = SystemTimeProvider.INSTANCE;
   }
 }

Before calling the YamlTester.testMessage() method, we set the TimeProvider to an instance of the SetTimeProvider class. This ensures that whenever the Chronicle Services runtime makes a call to retrieve the current time, the same value will always be returned. Notice the value we use in the call is the same as the timestamp in the expected output event. This ensures that the comparison will always succeed.

Once the tests have completed the TimeProvider is reset to the default value, which passes methods through the underlying system.

Replacing Expected Output With Actual Output

Sometimes changes to the component under test will result in the expected output in all test cases requiring to be changed. This could  potentially be a labour intensive and error prone task. Fortunately, YamlTester has a feature that allows it to perform what is known as “test regressions”, in which the actual output for each test is written back into the “expected output” file out.yaml.

$ mvn test -Dregress.tests=true

This will overwrite the erroneous “expected data”, so some care should be taken before running the test with the regress option enabled. It is intended to be used when a bug is found that, when fixed, will cause output to change in a way that would cause existing tests to fail.

Nevertheless, since the test data is part of the main source of the component (or at least it should be), when such a change is made and committed, previous versions of the component will still contain the version of test data that worked with that version.

One interesting use case for this is where comments are added to a number of tests after they have been shown to be functionally correct. Running the test suite with the regress.tests option enabled will generate the output with the comments in place.

Conclusion

Chronicle Services allows tests to be written and applied using a data-driven approach following the guidelines of Behaviour Driven Development. Features such as the regress.tests option allow large numbers of tests to be maintained with lower effort than other alternatives. The intention of this is to bring testing more effectively into the main development process, increasing the overall quality of components and applications using Chronicle Services.