mlsanityData to 14 Oct 2026

Asymmetric Virtual Memory Paging for Hybrid Mamba-Transformer Inference

arXiv 2605.22416 ↗First posted 21 May 2026cs.LG cs.DC cs.PFcited by 0

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Hybrid language models like Jamba mix attention layers with State Space Models (SSMs), creating two memory cache types with opposite profiles: Key-Value (KV) caches grow linearly with sequence length, while SSM states stay fixed per layer. Current inference engines handle this poorly. Unified pools pad SSM states to attention page sizes, wasting up to 7.3x capacity. Static dual pools cannot adapt when prompt distributions shift between requests. We present Asymmetric Virtual Memory Paging (AVMP). The allocator separates the two cache types into physically distinct pools behind a unified virtual address space, and migrates capacity between pools when one runs out. Migration triggers only on allocation failure, keeping behavior deterministic. We evaluate AVMP across 270 synthetic cells plus 60 cells of ShareGPT trace replay on an RTX 3060 12GB. Out-of-Memory events drop 7.6% and request throughput improves 1.83x to 13.3x across synthetic workloads and 2.36x on ShareGPT. All gains hold under paired-bootstrap 95% confidence intervals. A phase-time breakdown reveals two distinct mechanisms: shorter OOM recovery on capacity-pressured workloads, and faster allocation calls on KV-heavy workloads. Implementation is pure Python; Triton integration is future work.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 11 of its 21 references that are in the map. First posted on arXiv 21 May 2026.

Builds on

11 of 21 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling11 Jun
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec

Other references · 8

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  6. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  7. 2024RecurrentGemma: Moving Past Transformers for Efficient Open Language Modelscited by 64
  8. 2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Servingcited by 15

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
  2. 2025Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
  3. 2026MiniPIC: Flexible Position-Independent Caching in <100LOC
  4. 2026Comparative Characterization of KV Cache Management Strategies for LLM Inference
  5. 2025Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference
  6. 2025A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs
  7. 2026Efficient, VRAM-Constrained xLM Inference on Clients
  8. 2025Stateful KV Cache Management for LLMs: Balancing Space, Time, Accuracy, and Positional Fidelity