Ray Serve's LLM serving stack now has a routing option that looks at the number of tokens in each request before deciding where to send it. The new token-load-aware routing balances compute load across worker nodes while also factoring in the KV cache state on each one. That means a request could be sent to a busier server if that server already has the relevant cache to reuse.
Why token counts matter
In large language model inference, tokens are the unit of work. A request with a long prompt or a long generation requires more compute and more memory. The KV cache is a per-request state that stores the key and value tensors from the attention layers, so that generating the next token doesn't require reprocessing the entire previous context. When a server already has a cache that matches part of a new request, it can skip that work entirely. But that cache is only useful if the request ends up on that server.
Traditional routing that simply spreads requests evenly across nodes ignores this cache opportunity. It also ignores the fact that some requests are far heavier than others. A round-robin or queue-length approach can overload a node with a few large requests while other nodes sit idle with warm caches.
Balancing compute and cache reuse
The new mechanism in Ray Serve LLM weighs both dimensions. It considers the token load of each incoming request and the current compute load on each worker. Then it checks the KV cache state on the workers. If one worker has a cache that closely matches the incoming request, that request gets a bonus to be routed there. If the worker is already under a heavy compute load, the routing algorithm may still send it elsewhere to avoid a spike.
The balancing act is real. Sending a request to a node with a matching cache saves a cold start, but it concentrates traffic on that node. Sending it to a less loaded node keeps the overall pool more even but throws away the cache. The token-load-aware routing is designed to make that decision dynamically, per request, rather than relying on a static policy.
For teams running big model services, the routing layer is where latency and throughput are won or lost. A request that misses the cache can double the effective compute time. A node that gets slammed with long requests can become a bottleneck. The new feature gives operators another control knob when tuning their serving infrastructure.
Ray Serve is the serving layer for the Ray ecosystem, handling request distribution, scaling, and fault tolerance. The LLM-specific routing is an addition to that core, aimed at transformer workloads that are memory and compute intensive.
How much the token-aware routing helps will depend on the traffic pattern. Workloads with many similar prompts will see more cache hits, while workloads with wildly varying request lengths might require careful tuning of the cache and load weights. The routing configuration is expected to be adjustable, allowing operators to prioritize cache reuse or even load depending on their own service-level goals. The first step is to measure what the default setting does with real traffic, then tweak from there.
