AI guides

FlashStates jointly manages recurrent state and KV caches for hybrid model inference

By 机器之心2 viewsSource: 机器之心
FlashStates hybrid recurrent-state and KV-cache management

FlashStates manages recurrent state and KV caches together for hybrids using KDA, GDN and Mamba-style layers. The team tests prefix reuse, prefill speed and tiered-cache benefits on B200 GPUs with long-context workloads.

Editorial correction (November 17, 2026): TTFT ratio wording has been corrected: 7.18 seconds is about one-third of the listed controls, not a “threefold reduction in time.” Benefits apply only to the tested conditions.

Article illustration

Linear-attention and state-space layers such as Kimi Delta Attention (KDA), Gated DeltaNet (GDN) and Mamba-2 are becoming common in open models. Kimi K3.1 Pro, GLM-5.3-Flash, Qwen3.8-Flash-Next and Nemotron 3 Super combine full attention with linear layers for faster decoding and smaller KV caches.

But prefix caching is more complex than in pure Transformers for long-context, multi-turn coding agents. Together AI, UIUC and UT Austin propose FlashStates to redesign joint KV/state management. Replaying real Claude Code trajectories on Nvidia B200 GPUs, they report up to 3.14 times vLLM’s prefill throughput and 88.4% lower p90 time to first token (TTFT). Gains persist with a 512 GiB Mooncake CPU cache, with distributed storage and cross-node hybrid-state reuse supported.

Article illustration

  • Paper: FlashStates: Unlocking Flexible Hybrid State Management for Agentic LLM Inference
  • Paper link: https://www.alphaxiv.org/abs/2611.FlashState
  • Code: https://github.com/FutureMLS-Lab/FlashStates
  • Project: https://FlashStates.ai

Authors

This work is a collaboration between Together AI, the University of Illinois Urbana-Champaign and the University of Texas at Austin. Co-first authors are Chenxi Li (Together AI) and Weili Xu (UIUC), with Tianyi Zhang, Jue Wang and Zhongzhu Zhou (Together AI) and corresponding author Chenfeng Xu (UT Austin).

Together AI was founded in June 2022 by Flash Attention author Tri Dao, Stanford foundation-model research center director Percy Liang, University of Chicago associate professor Ce Zhang and former Apple executive Vipul Ved Prakash. FutureMLS, Future Machine Learning & Systems Lab, is a nonprofit founded by University of Sydney computer-science PhD Zhongzhu Zhou, researching efficient ML architectures, post-training, training/inference engines and agent systems.

Hybrid state: two indispensable parts

Pure Transformers store KV per token, allowing reuse at any prefix cutoff. Linear layers summarize prefixes in fixed-size recurrent states. These do not grow with prompt length, but individual state blocks are large and cannot be rolled back: state at token 10000 cannot recover state at token 9000.

Therefore, continuing computation from position p requires both of the following: 1. the attention KV for the complete prefix, and 2. a recurrent checkpoint located exactly at p, referred to below simply as a checkpoint. The paper calls these two components together the 'hybrid state.'

Each agent round includes the previous prompt, output and tool results. The ideal reuse point is the previous decode endpoint, which usually does not align with preset block boundaries.

Existing engines: three kinds of misses

Running GLM-5.3-Flash in vLLM and SGLang, the team traced recomputed prompt tokens to three avoidable misses: placement, where a checkpoint was not saved at the needed position; dependency, where KV remains but its checkpoint was evicted; and capacity, where the cache genuinely cannot hold everything.

Article illustration

Attributing cache misses in vLLM and SGLang

By default, vLLM saves checkpoints only at 4,352-token prompt-block boundaries, not during decoding. At concurrency eight to sixteen, placement misses account for 43% of uncached tokens. SGLang supports finer boundaries, such as 256 tokens, reducing placement misses below 5% at the same concurrency. But roughly 430 checkpoints compete in an independent pool; LRU eviction often removes checkpoints whose KV survives, causing dependency misses.

At concurrency 32–64, problems compound. vLLM returns prompt-end checkpoints to the evictable queue after decoding begins. Dependency misses rise rapidly from concurrency forty and account for 42% of uncached tokens at sixty-four. Both engines also begin hybrid-state thrashing around forty: capacity misses add to dependency misses.

The conclusion is that making checkpoint intervals too large produces placement misses, making them too small produces dependency misses, and higher concurrency adds capacity misses on top of those. Adjusting checkpoint density alone cannot solve these problems.

FlashStates design

1. Arbitrary-position State Checkpointing (ASC).

At request completion, FlashStates captures state at the exact endpoint. If inside a KV block, it retains the whole final block and jointly manages full-attention KV and recurrent checkpoints. For mid-prefill positions, PrefillExport lets kernels export selected states during normal computation without changing outputs. KDA and GDN kernels are adapted.

2. Define the objective

With ASC, hybrid states can exist at arbitrary positions. The remaining question is which ones to retain when GPU memory cannot hold them all. The paper uses two positions to describe how much computation a request r wastes (see the figure below). The ideal recovery point, ℓ(r)(*), is the longest position up to which the request's prompt matches any previously processed content; with a sufficiently large cache, computation could resume there. The actual recovery point, ℓ(r)(π), is the latest position at which a complete hybrid state remains in the cache under policy π. Tokens between these two positions must undergo prefill again, and the time this takes is denoted C(ℓ(r)(π), ℓ(r)(*)).

The objective: under GPU-memory budget B, bring actual ℓ(r)(π) as close as possible to ideal ℓ(r)(*), minimizing total repeated-prefill time Σ(r) C(ℓ(r)(π), ℓ(r)(*)).

Article illustration

3. Dependency-aware admission and eviction.

An online policy based on GreedyDual-Size (GDS) assigns each state priority H(u) = watermark at last use + recovery cost ÷ blocks actually freed by eviction. Here, recovery cost is the additional prefill time that related requests must incur after eviction, estimated using the cost model above. The eviction unit is one checkpoint plus the blocks referenced exclusively by that checkpoint; prefixes shared with other hybrid states do not count toward the amount of memory released. Both terms are measured in prefill time per block, so they can be added directly without any hyperparameter that must be adjusted to the workload. Admission uses this same priority: when GPU memory is tight, a new checkpoint is retained only if it is more valuable than the least valuable checkpoint already cached.

GPU-cache experiments

Tests use B200 GPUs and SemiAnalysis’s public Claude Code trajectories. For GLM-5.3-Flash with 1M context and 48 sessions:

Article illustration

Across three models, prefill throughput is 3.15 times vLLM for GLM-5.3-Flash, 2.27 for Nemotron 3 Super and 1.54 for Qwen3.8-Flash-Next. Corresponding p90 TTFT reductions are 88.4%, 77.1% and 79.0%.

Article illustration

Serving performance versus concurrency across hybrid models

Beyond GPU: CPU caches and cross-engine reuse

In production, inference services usually offload cache that does not fit in GPU memory to CPU memory, or even to shared storage. If the cache becomes sufficiently large, do FlashStates' benefits remain? The paper's answer is that a larger cache can retain more data, but cannot provide a checkpoint that was never saved.

The team integrated Mooncake as a CPU cache with three changes:

  • Asynchronous write-through at completion: new exact-endpoint state queues for CPU copying immediately, rather than waiting for GPU eviction;
  • Unified two-level management: Mooncake metadata tracks ownership and shared-KV references, extending dependency-aware GDS across GPU and CPU and preventing deletion of still-needed blocks;
  • Transfer optimization: improved GPU-to-CPU copies and compressed states associated with speculative MTP decoding.

Tests use GLM-5.3-Flash NVFP4, 1M context, two B200s per service and three-step MTP, comparing vLLM plus 512 GiB Mooncake and SGLang plus 511.5 GiB HiCache:

  • Output throughput improves up to 41.4% over vLLM/Mooncake and 36.2% over SGLang/HiCache.
  • Median uncached prompt tokens fall 68.7–81.4%.
  • At eighty sessions, p90 TTFT is 7.18 seconds versus 21.88 and 22.97, about one-third of the controls.

Article illustration

Input throughput and decoding speed with a 512 GiB CPU cache; bubble area indicates p90 TTFT. GLM-5.3-Flash NVFP4, 32–80 sessions.

States in shared storage can further be reused by subsequent rounds on another engine worker, provided the prefixes match exactly and all objects required by every tensor-parallel (TP) rank are present. This lays the groundwork for distributed deployments such as prefill–decode (PD) disaggregation, where the next round can be taken over by another prefill engine.

Conclusion

FlashStates' central argument is that recovery positions should be determined by where reuse is possible, rather than by how KV is divided into blocks. On the engineering side, it has been integrated with vLLM, covers three hybrid architectures, is compatible with CPU caching and speculative decoding, and prepares for state sharing across engines. As hybrid architectures become more common and agent contexts continue to grow, inference-system caching logic designed for pure Transformers needs to be redesigned for hybrid states. The open-source inference engine SGLang has now adopted the cost-aware state-eviction policy proposed by FlashStates and explicitly cites this work in its official documentation.

Reference: https://docs.sglang.io/docs/advanced_features/radix_eviction_policy#state_cost-parameters

This article was originally published by Synced. Author: Synced.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)