Using Pausers in Event Loops

Typically in low-latency development, a trade-off must be made between minimising latency and avoiding excessive CPU utilisation. While the simplest approach in many Java programs is to call Thread.sleep and hope for the best, that strategy can easily place the main thread at the mercy of the operating system scheduler, extending the intended sleep time and introducing jitter. This article explores how Chronicle’s Pausers can be used to automatically apply a back-off strategy when there is no data to be processed, providing an excellent balance between resource usage and responsive, low-latency, low-jitter applications, and doing so in a far more deterministic way than a naïve thread sleep.

Description of the Problem

In a typical application stack multiple threads are used for servicing events, processing data, pipelining etc., forming the backbone of modern Java concurrency patterns. An important design consideration is how threads become aware that there is work to do, with some general approaches including:

• Signal/Notification: In this case the receiving thread yields (is added to a wait queue) until notified by another thread. This has the benefit of low resource consumption; however, there is a relatively high latency of at least 20-50 microseconds (and likely much more – see below) to reschedule the thread in response to a signal, an effect you will notice immediately if you profile view posts on Stack Overflow discussing “why is my Thread.sleep so slow?”.

• Busy Waiting: In this case the receiving thread continually spins checking for some indication that there is work to do. This has the benefit of quick response (low latency) when there is work for the thread to do; however, it comes at the expense of high CPU usage, wasting cycles when there is nothing to do. In addition, the constant high CPU usage in turn leads to appreciably higher power demand and associated cooling load, something that even bronze badges and silver badges–level performance tuners sometimes overlook.

• Fixed Sleep: In this case, when there is no further work to be done the receiving thread sleeps for a fixed period of time before checking again for more work. This has the benefit of low resource usage, but the clear downside of this strategy is that worst-case latency is at least as large as the sleep period. A single mis-calculated sleep time on your main thread can therefore push tail-latency well beyond your SLA.

The Problems with Sleeping, and how to Sleep Soundly

The actual behaviour when a thread requests a sleep varies not only across platforms, but also across different versions and usage patterns for the same platform. Java’s sleep method specifies that the current thread sleep for at least the requested period, but nothing in the contract guarantees upper bounds on wake-up latency.

For example, POSIX requires that sleep calls always yield the CPU, whereas Linux allows sleep implementations (including sleep, usleep, nanosleep and similar) to busy wait in some cases. For older versions of Linux with fixed timer ticks (usually 100 Hz, 250 Hz, or 1000 Hz) there was a relatively large penalty when yielding to the scheduler, which encouraged the use of busy-waiting internally within sleep calls for short periods. By contrast, more recent versions of Linux have more sophisticated schedulers using dynamic ticks, which enable more accurate short-period interactions with a sleeping thread, largely removing the need for busy-waiting to achieve low sleep periods. That said, seasoned engineers still need to catch InterruptedException and handle unexpected wake-ups gracefully.

The following rules of thumb generally apply across recent Linux versions for standard processes (i.e. those running with normal permissions under the standard scheduler):

• sleep requests of approximately 1 µs can in principle be serviced with reasonable accuracy, although you should always test this in the context of your own java concurrency scenario.

• In general, even short sleep periods will not busy wait – although extremely short periods almost certainly will.

• sleep requests of ~1 ms and ~1 µs reduce CPU usage to ~1 % and ~10 % respectively compared with busy waiting (100 %).

While the above suggests that even relatively short sleeps of ~1 µs could potentially provide a useful compromise between latency and resource use, the major issue is scheduling: as soon as the sleeping process is fully context switched off a core, the overhead to reschedule can be orders of magnitude higher than the intended sleep period. If you check any high-profile view user profile of low-latency engineers, you will see numerous war stories of Thread.sleep(0) leading to millisecond stalls.

Here again, there is no single answer as to how the system will behave. The key is to bias the situation as much as possible to avoid the thread being switched from a core, and the use of thread affinity (to avoid the thread being moved to another core) and CPU isolation (to avoid another process/thread contending with the thread) can be very effective in this case¹. Careful use of affinity, isolation and short sleep periods can result in responsive, low-jitter environments, which use considerably fewer CPU resources compared with busy waiting. In practice, combining these techniques with an adaptive back-off such as Chronicle’s Pauser enables you to pause Java threads predictably without sacrificing latency.

¹ Other options include running with real-time priorities; however, we want to keep the focus of this document on standard setups as much as possible.

What are Pausers?

Chronicle’s Pausers provide a sliding scale of behaviours between the above extremes of Signal/Notification, Fixed Sleep and Busy Waiting, by using an intelligent back-off strategy which enables a more nuanced control to better balance low latency and resource utilisation. From a code-level perspective, you can think of a Pauser as a smarter wrapper around Thread.sleep, sleep thread yields and spin-loops, abstracting away the decision of when to let the current thread sleep and when to keep it on-CPU.

The general strategy is to busy-wait for a short period before incrementally backing off to longer and longer pauses (consuming decreasing amounts of CPU) when there is no work to be done. Different strategies (Pauser Modes) are available depending on the task, with the canonical way of using a Pauser being:

Pauser Modes

This table illustrates several different Pauser modes, as well as the benefits and downsides to using each of them.

Chronicle Pausers allow for optimising the CPU load for a given level of responsiveness and latency. This trade-off can be configured with high accuracy, without needing to make significant changes to your application code. For instance, if you realise that a particular thread needs to be more responsive, you can change its Pauser from a back-off Pauser to a busy Pauser and vice versa, rather than sprinkling additional Thread.sleep calls throughout the codebase.

Of note, the Busy Pauser used for lowest latency internally uses busy waiting and, as such, will consume 100 % of one core. It is therefore important to ensure Busy Pausers do not contend for the same core, and CPU affinity and isolation should be considered when using Busy Pausers to control this aspect. If you operate in a Spring Boot microservice environment, remember that container orchestration can move your static void main processes unpredictably, so explicit pinning of the main thread may be required.

Performance of Pauser Modes

The graph below plots the time waiting for an event (x-axis) against the pause/response time for a selection of Pausers.

The Busy, TimedBusy, Yielding and Millis Pausers show flat response times regardless of how long the thread waits to receive an event, but with varying response times due to the different yielding strategies vs CPU usage. In many cases TimedBusy in particular provides an excellent compromise between low latency and CPU usage, frequently matching the best manual configurations seen in high-scoring answer questions on community forums.

The Sleepy and Balanced strategies show step changes and steady growth in response times reflecting the incremental back-off the longer the thread waits to receive an event. These strategies also demonstrate how the library transparently transitions from spin to yield to java sleep without the developer having to micromanage sleep time parameters.

Conclusion

This article explored the use of Chronicle’s Pausers and how they are used to build responsive, low-latency, low-jitter applications with relatively low CPU utilisation. By delegating the decision of when the current thread sleep should occur, Pausers help you avoid the accidental latency spikes that arise from poorly chosen Thread.sleep durations. This in turn helps maximise hardware utilisation, while also reducing power consumption, helping reduce costs to your organisation and freeing developers to improve answers and add comments to their code rather than endlessly tuning spin loops.