Java Virtual Threads and Scaling

Java Virtual Threads were introduced in JDK 19 as a preview feature as part of Project Loom, but they are fully supported starting from JDK 21 and beyond (JDK 24 includes further refinements). 

Virtual threads primarily help developers by simplifying concurrent programming — eliminating the need to choose between synchronous and reactive models, and reducing the complexity of working with NIO (non-blocking I/O) APIs. They seamlessly translate traditionally blocking operations such as I/O calls, synchronization, and thread sleeping into efficient, non-blocking operations under the hood, while allowing developers to write code in a straightforward, blocking style.

Traditional Threading Model

In traditional multithreaded Java applications, the container creates N Java threads, with each Java thread mapped directly to an underlying OS thread. The operating system uses time-slicing to schedule these OS threads for execution on a limited number of CPU cores. When a thread performs a blocking operation, such as an I/O call, acquiring a lock (synchronization), or calling sleep(), the associated OS thread is taken off the CPU, and another thread is scheduled in its place. 

However, when there are too many threads competing for CPU time, excessive context switching occurs, leading to performance degradation due to the overhead of constantly saving and restoring thread states. 

Java Virtual Threading Model

In the Java Virtual Threading model, the JVM utilizes a small number of carrier threads, typically equal to the number of available CPU cores. Each carrier thread is backed by a single OS thread, meaning that there is minimal context switching at the OS level. Instead of relying on the operating system, the JVM itself handles the scheduling and context switching of virtual threads entirely in user space.

When a Java application creates N virtual threads, the JVM efficiently maps them onto the available carrier threads. These virtual threads are extremely lightweight compared to traditional platform threads, allowing the application to spawn thousands (or even millions) of them without significant resource overhead. The JVM manages the lifecycle and execution of virtual threads internally, suspending and resuming them as needed without involving the OS scheduler, which greatly enhances scalability and performance for concurrent applications.

Thread Pool and CPU Scheduling

In a typical business application, workloads generally consist of both CPU-bound operations (such as calculations or data processing) and I/O-bound operations (like remote service calls or database interactions) to support the required functionality.

To achieve higher concurrency and better utilization of system resources, applications often define a thread pool with a size greater than the number of available CPU cores. This allows threads that are blocked on I/O to give way to others that can continue executing, ensuring the CPUs remain busy.

Now, consider a CPU-bound business operation, such as computing the factorial of large numbers. Suppose we need to complete 20 such operations, each taking approximately 1 second to execute on a machine with 4 CPU cores.

Traditional Threads

With traditional threads, the operating system uses time-slicing to share CPU cores among multiple threads. In this case, if we create 20 threads to perform 20 CPU-bound operations on a 4-core machine, the OS will schedule them in a round-robin fashion, giving each thread a time slice (e.g., 100ms) on the available cores.

As a result, instead of processing four operations at a time (as in an ideal parallel execution scenario), all 20 threads will be interleaved across the CPU cores. Each thread gets a short time slice, waits, and then gets scheduled again. This continues until each operation completes. 

Due to the constant context switching and shared CPU time, each individual operation will still take approximately 5 seconds to complete, even though all operations may appear to finish around the same time. This leads to increased latency per operation and inefficient CPU utilization in CPU-bound scenarios.

Virtual Threads

Virtual threads do not rely on OS-level time slicing. Once a virtual thread is mounted on a carrier thread, it continues executing uninterrupted until it hits a blocking operation (e.g., I/O, synchronization). In a purely CPU-bound workload, like computing factorials, each virtual thread will keep running as long as it has CPU time available.

On a 4-core machine, this means that only four virtual threads will run at a time, each fully utilizing a CPU core. The first four operations will complete in approximately 1 second, the next 4 in 2 seconds, and so on. By the end of 5 seconds, all 20 operations will be completed.

The key distinction here is that, while the total execution time remains the same as with traditional threads (because we’re limited by CPU capacity), individual operations finish sooner, and context switching overhead is reduced. This results in more predictable performance and better CPU utilization, especially when mixing CPU-bound and I/O-bound workloads.

CPU-Bound Operation Processing Time Comparison

Below are the results of (2000000000!) compute factorial operation on 8 CPU AMD EPYC 9J14 96-Core Processor, CPU MHz 2596. 200 operations computed.

• Each operation (2000000000!) takes ~1.68s.
• Traditional thread with 200 pool size takes an average of 35s for each operation.
• Virtual thread takes ~1.68s.

Final Thought

Java virtual threads simplify development by eliminating the need for complex non-blocking reactive programming or manual tuning of thread pool sizes. Since applications can be written in a straightforward, synchronous style, developers can easily trace execution using standard stack traces in log files, greatly improving observability and debugging. 

Virtual threads also offer improved throughput and faster response times for concurrent, CPU-bound operations, thanks to reduced context-switching overhead and efficient scheduling managed by the JVM.