mlsanityData to 14 Oct 2026

MELODI: Exploring Memory Compression for Long Contexts

arXiv 2410.03156 ↗First posted 4 Oct 2024cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

We present MELODI, a novel memory architecture designed to efficiently process long documents using short context windows. The key principle behind MELODI is to represent short-term and long-term memory as a hierarchical compression scheme across both network layers and context windows. Specifically, the short-term memory is achieved through recurrent compression of context windows across multiple layers, ensuring smooth transitions between windows. In contrast, the long-term memory performs further compression within a single middle layer and aggregates information across context windows, effectively consolidating crucial information from the entire history. Compared to a strong baseline - the Memorizing Transformer employing dense attention over a large long-term memory (64K key-value pairs) - our method demonstrates superior performance on various long-context datasets while remarkably reducing the memory footprint by a factor of 8.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

27 of 36 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
15 more
  1. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  3. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  4. 2023Adapting Language Models to Compress Contextscited by 356
  5. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  6. 2022Memorizing Transformerscited by 226
  7. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  8. 2022Recurrent Memory Transformercited by 203
  9. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  10. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  11. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  12. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  13. 2022Block-Recurrent Transformerscited by 110
  14. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 105
  15. 2024Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?cited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 5.

11 papers.

  1. 2024Titans: Learning to Memorize at Test Timecited by 328
  2. 2025Memory in the Age of AI Agentscited by 303
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  4. 2025R3Mem: Bridging Memory Retention and Retrieval via Reversible Compressioncited by 12
  5. 2025Mind Meets Space: Rethinking Agentic Spatial Intelligence from a Neuroscience-inspired Perspectivecited by 7
  6. 2026The Topological Trouble With Transformerscited by 3
  7. 2025LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMscited by 2
  8. 2026Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoningcited by 2
  9. 2026Towards Compressive and Scalable Recurrent Memorycited by 1
  10. 2026Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compressioncited by 0
  11. 2026Kwai Summary Attention V2 Technical Reportcited by 0