# LoMA: Lossless Compressed Memory Attention

arXiv: https://arxiv.org/abs/2401.09486 · HTML: https://mlsanity.com/paper/2401.09486/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 11 of its 15 references that are in the map.
- First posted on arXiv 16 Jan 2024.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 11 of 11)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Learning to Compress Prompts with Gist Tokens](https://mlsanity.com/paper/2304.08467/) (2023)
- [LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models](https://mlsanity.com/paper/2310.05736/) (2023)
- [SparQ Attention: Bandwidth-Efficient LLM Inference](https://mlsanity.com/paper/2312.04985/) (2023)
- [Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection](https://mlsanity.com/paper/1912.11637/) (2019)
- [Memory-efficient Transformers via Top-k Attention](https://mlsanity.com/paper/2106.06899/) (2021)
- [Learning to Reason and Memorize with Self-Notes](https://mlsanity.com/paper/2305.00833/) (2023)

## Built on by (most cited first, 6 of 6)

- [A Survey on Large Language Model Acceleration based on KV Cache Management](https://mlsanity.com/paper/2412.19442/) (2024)
- [Designing Large Foundation Models for Efficient Training and Inference: A Survey](https://mlsanity.com/paper/2409.01990/) (2024)
- [SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference](https://mlsanity.com/paper/2508.02751/) (2025)
- [From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.12904/) (2026)
- [Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies](https://mlsanity.com/paper/2603.27960/) (2026)
- [Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression](https://mlsanity.com/paper/2609.36322/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) face limitations due to the high demand on GPU memory and computational resources when handling long contexts. While sparsify the Key-Value (KV) cache of transformer model is a typical strategy to alleviate resource usage, it unavoidably results in the loss of information. We introduce Lossless Compressed Memory Attention (LoMA), a novel approach that enables lossless compression of the KV cache, thereby reducing the memory and computational demands during autoregressive generation. LoMA incorporates a specialized training or fine-tuning precedure alongside an autoregressive generation algorithm optimized for the compressed context. Our method compresses the KV cache after every tc generated tokens with a compression ratio of c and a target compressed length t, and this process occurs within a single inference pass without dependency on auxiliary models. We engineered an efficient training scheme involving specific inputs, attention masks, and position identifiers to instill this compression capability. Experimental validation has demonstrated that LoMA significantly reducing computational consumption and memory usage through achieving lossless KV cache compression.