mlsanityData to 14 Oct 2026

Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving

arXiv 2606.20537 ↗First posted 18 Jun 2026cs.LG cs.DCcited by 7

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Mainstream LLM serving systems reuse prefix work mainly through paged or radix key-value (KV) caches. This is highly effective for high-throughput, high-concurrency serving, but it manages only one positional fragment of execution state: the KV cache. We study the opposite regime: low-latency, small-batch, on-device physical-AI serving, where interactive LLM agents, speech systems, and robot policies repeatedly branch, reset, interrupt, and re-enter under tight responsiveness budgets. We introduce execution-state capsules, a graph-bound checkpoint and restore mechanism for the complete restorable state at a committed boundary. FlashRT is a white-box, backend-facing kernel runtime whose evaluated NVIDIA CUDA backend runs captured graph plans over contiguous static buffers with no block-table indirection. Because the live state is a closed set of named buffers, a capsule can snapshot, restore, fork, or roll back the whole execution boundary, including KV, recurrent state, convolution state, MTP state, and metadata. This moves reuse from token-addressed KV fragments to graph-bound execution-state boundaries. On an RTX 5090, capsule restore is byte-exact at the stored-state level and token-identical under greedy decode. A KV-only ablation diverges, showing that recurrent state is load-bearing. GPU-resident snapshot and restore are sub-millisecond, and TTFT speedup over cold prefill grows from 3.9x at 2k tokens to 27x at 16k tokens. On Jetson AGX Thor and DGX Spark, the same correctness and structural properties hold. Capsules are not a replacement for high-throughput KV-cache serving; they define a complementary latency-first serving point for explicit execution-state reuse.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 12 of its 16 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

12 of 16 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention7 May

Other references · 11

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2022Flow Matching for Generative Modelingcited by 5,702
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  8. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  9. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  10. 2024Hydragen: High-Throughput LLM Inference with Shared Prefixescited by 73
  11. 2023Stateful Large Language Model Serving with Pensievecited by 61

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Jetson-PI: Towards Onboard Real-Time Robot Control via Foresight-Aligned Asynchronous Inferencecited by 3
  2. 2026Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robotscited by 2
  3. 2026PhyAI: Real-Time Physical AI at the Edge, Scalable Rollouts in the Cloudcited by 1
  4. 2026Stateful Worlds, Stateless Elasticity: Exact-State Serving for Interactive World Modelscited by 0
  5. 2026Aborted but Not Forgotten: KV-Cache Retention Breaks Rollback Consistency in Language Agentscited by 0
  6. 2026Efficient Vision-Language-Action Management and Serving for Robot Factoriescited by 0
  7. 2026rMuscle: Robotic Muscle Memory for Efficient Vision-Language-Action Model Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Concordia: JIT-Compiled Persistent-Kernel Checkpointing for Fault-Tolerant LLM Inference
  2. 2026Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs
  3. 2024Fast State Restoration in LLM Serving with HCache
  4. 2026GhostServe: A Lightweight Checkpointing System in the Shadow for Fault-Tolerant LLM Serving
  5. 2026Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents
  6. 2026CacheFlow: Efficient LLM Serving with 3D-Parallel KV Cache Restoration
  7. 2026Bounded-State Restoration: Decoupling Local Restore Capacity from External LLM State
  8. 2026PrefixBench-H100: Characterizing Prefix Reuse and Time-to-First-Token in H100 LLM Serving