mlsanityData to 7 Oct 2026

LoMA: Lossless Compressed Memory Attention

arXiv 2401.09486 ↗First posted 16 Jan 2024cs.LG cs.CLcited by 5

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large Language Models (LLMs) face limitations due to the high demand on GPU memory and computational resources when handling long contexts. While sparsify the Key-Value (KV) cache of transformer model is a typical strategy to alleviate resource usage, it unavoidably results in the loss of information. We introduce Lossless Compressed Memory Attention (LoMA), a novel approach that enables lossless compression of the KV cache, thereby reducing the memory and computational demands during autoregressive generation. LoMA incorporates a specialized training or fine-tuning precedure alongside an autoregressive generation algorithm optimized for the compressed context. Our method compresses the KV cache after every tc generated tokens with a compression ratio of c and a target compressed length t, and this process occurs within a single inference pass without dependency on auxiliary models. We engineered an efficient training scheme involving specific inputs, attention masks, and position identifiers to instill this compression capability. Experimental validation has demonstrated that LoMA significantly reducing computational consumption and memory usage through achieving lossless KV cache compression.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 11 of its 15 references that are in the map. First posted on arXiv 16 Jan 2024.

Builds on

11 of 15 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec

Other references · 10

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  5. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  6. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  7. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  8. 2019Explicit Sparse Transformer: Concentrated Attention Through Explicit Selectioncited by 61
  9. 2021Memory-efficient Transformers via Top-k Attentioncited by 59
  10. 2023Learning to Reason and Memorize with Self-Notescited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 1; 2026: 2.

5 papers.

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  2. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  3. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 5
  4. 2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generationcited by 5
  5. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy
  2. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  3. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs
  4. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference
  5. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  6. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  7. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  8. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs