mlsanityData to 7 Oct 2026

Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving

arXiv 2607.02640 ↗First posted 2 Jul 2026cs.SD cs.AI cs.NIcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Real-time interaction models -- Moshi, MiniCPM-o, Qwen-Omni -- turn serving into a periodic real-time task: on every frame a session ingests streaming audio and must respond by a recurring wall-clock deadline, while its KV cache grows monotonically and stays pinned for the whole conversation. This regime hides a dangerous failure mode. On a real full-duplex stack, sustained load does not degrade serving gracefully: it falls off a cliff, jumping in one step from milliseconds per frame to a stalled engine when accumulated session state exhausts the KV pool. The collapse is metastable -- identical five-minute runs collapse or survive on run-to-run variance -- and silent: latency and deadline-miss metrics read healthy throughout. We show one move restores both stability and observability: bound each session's resident state, and latency starts telling the truth. Metronome's in-engine KV window eliminates the collapse (0/20 vs. 14/20 runs across two batches) and turns per-frame latency into a monotone load signal, on which an online admission controller discovers the schedulable concurrency; without the window, the identical controller over-admits into the wall. A first-order model predicts the collapse time within a few percent on the headline model, and a quality probe validates the bound's design by ablation: the window alone is quality-free in turn-based decoding, and its few pinned attention-sink tokens are what keep free-running generation healthy. Everything is measured end-to-end on real audio, across four interaction models on one GPU.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 36 of its 46 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

36 of 46 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  2. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun
  5. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time26 May

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2023Mistral 7Bcited by 3,065
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
19 more
  1. 2023Neural Codec Language Models are Zero-Shot Text to Speech Synthesizerscited by 737
  2. 2025Qwen2.5-Omni Technical Reportcited by 704
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  4. 2022High Fidelity Neural Audio Compressioncited by 673
  5. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  6. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  9. 2024Qwen2-Audio Technical Reportcited by 543
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  11. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  12. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  13. 2025Qwen3-Omni Technical Reportcited by 451
  14. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 204
  15. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  16. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  17. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  18. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 64
  19. 2023AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Servingcited by 31

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Tangram: Unlocking Non-Uniform KV Cache Compression for Efficient Multi-turn LLM Serving
  2. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
  3. 2025FlexiCache: Leveraging Temporal Stability of Attention Heads for Efficient KV Cache Management
  4. 2026InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systems
  5. 2025Mitigating KV Cache Competition to Enhance User Experience in LLM Inference
  6. 2026Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving
  7. 2026Characterizing High Bandwidth Flash for LLM Serving
  8. 2026Characterizing Contention-Induced Reliability Collapse in KV-Cache Timing Side Channels for Multi-Tenant LLM Serving