mlsanityData to 7 Oct 2026

RW-TTT: Batched Serving for Request-Owned Test-Time Training State

arXiv 2605.28053 ↗First posted 27 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Test-time training (TTT) adapts an LLM during generation by reading and updating request-owned state, such as fast weights, low-rank deltas, or streaming learner state. This breaks batched LLM serving, which assumes shared static weights: serial execution is correct but slow, while naive batching can corrupt request state. We formulate this problem as read-write TTT serving and present RW-TTT , which tags each decode step with its owner, version, and READ/WRITE effect, batches only compatible phases, and commits updates only to the owner. On one GPU with eight fast-weight InPlace-TTT streams, RW-TTT reaches 274.61 aggregate tok/s, 9.31x over sequential serving and 3.44x over per-stream replicas under the same memory budget. It preserves behavior on RULER, a long-context benchmark, and passes owner/version checks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 11 of its 23 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

11 of 23 references.

References · 11

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  3. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  6. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  7. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  8. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  9. 2025End-to-End Test-Time Training for Long Contextcited by 53
  10. 2024ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Servingcited by 5
  11. 2025Efficient LLM Serving on Hybrid Real-time and Best-effort Requestscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Test-Time Training Done Right
  2. 2026Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training
  3. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  4. 2026Rethinking Expressivity and Efficiency in Test-Time Training
  5. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling
  6. 2026Universal Test-Time Training
  7. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  8. 2026Beyond Prediction: Tail-Aware Scheduling for LLM Inference