mlsanityData to 14 Oct 2026

MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models

arXiv 2504.12526 ↗First posted 16 Apr 2025cs.LG cs.AI cs.CLcited by 0

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Long-context language models exhibit impressive performance but remain challenging to deploy due to high GPU memory demands during inference. We propose Memory-efficient Offloaded Mini-sequence Inference (MOM), a method that partitions critical layers into smaller "mini-sequences" and integrates seamlessly with KV cache offloading. Experiments on various Llama, Qwen, and Mistral models demonstrate that MOM reduces peak memory usage by over 50% on average. On Meta-Llama-3.2-8B, MOM extends the maximum context length from 155k to 455k tokens on a single A100 80GB GPU, while keeping outputs identical and not compromising accuracy. MOM also maintains highly competitive throughput due to minimal computational overhead and efficient last-layer processing. Compared to traditional chunked prefill methods, MOM achieves a 35% greater context length extension. More importantly, our method drastically reduces prefill memory consumption, eliminating it as the longstanding dominant memory bottleneck during inference. This breakthrough fundamentally changes research priorities, redirecting future efforts from prefill-stage optimizations to improving decode-stage residual KV cache efficiency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 27 of its 39 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

27 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading18 Feb
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  12. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
13 more
  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2020Reformer: The Efficient Transformercited by 1,668
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  9. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  10. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  11. 2021Self-attention Does Not Need O(n2) Memorycited by 154
  12. 2024Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoningcited by 110
  13. 2024Mini-Sequence Transformer: Optimizing Intermediate Memory for Long Sequences Trainingcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU
  2. 2025LLaMCAT: Optimizing Large Language Model Inference with Cache Arbitration and Throttling
  3. 2025A Cost-Effective Near-Storage Processing Solution for Offline Inference of Long-Context LLMs
  4. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inference
  5. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  6. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  7. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs
  8. 2025Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU