mlsanityData to 7 Oct 2026

MemLong: Memory-Augmented Retrieval for Long Text Modeling

arXiv 2408.16967 ↗First posted 30 Aug 2024cs.CL cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

Recent advancements in Large Language Models (LLMs) have yielded remarkable success across diverse fields. However, handling long contexts remains a significant challenge for LLMs due to the quadratic time and space complexity of attention mechanisms and the growing memory consumption of the key-value cache during generation. This work introduces MemLong: Memory-Augmented Retrieval for Long Text Generation, a method designed to enhance the capabilities of long-context language modeling by utilizing an external retriever for historical information retrieval. MemLong combines a non-differentiable ``ret-mem'' module with a partially trainable decoder-only language model and introduces a fine-grained, controllable retrieval attention mechanism that leverages semantic-level relevant chunks. Comprehensive evaluations on multiple long-context language modeling benchmarks demonstrate that MemLong consistently outperforms other state-of-the-art LLMs. More importantly, MemLong can extend the context length on a single 3090 GPU from 4k up to 80k. Our code is available at https://github.com/Bui1dMySea/MemLong

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 33 of its 40 references that are in the map. First posted on arXiv 30 Aug 2024.

Builds on

33 of 40 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
21 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  5. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  7. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  8. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  10. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  11. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  12. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  13. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  14. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  15. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  16. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  17. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  18. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  19. 2024LongHeads: Multi-Head Attention is Secretly a Long Context Processorcited by 16
  20. 2024Instruct Once, Chat Consistently in Multiple Rounds: An Efficient Tuning Framework for Dialoguecited by 11
  21. 2023TRAMS: Training-free Memory Selection for Long-range Language Modelingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 5.

11 papers.

  1. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  2. 2026M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactionscited by 20
  3. 2024Dynamic Planning for LLM-based Graphical User Interface Automationcited by 19
  4. 2025Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architecturescited by 9
  5. 2025NeSyPr: Neurosymbolic Proceduralization For Efficient Embodied Reasoningcited by 5
  6. 2026Advancing Multimodal Agent Reasoning with Long-Term Neuro-Symbolic Memorycited by 5
  7. 2025Cognitive Weave: Synthesizing Abstracted Knowledge with a Spatio-Temporal Resonance Graphcited by 3
  8. 2025Long Story Generation via Knowledge Graph and Literary Theorycited by 1
  9. 2026MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representationcited by 0
  10. 2026Narrative Aligned Long Form Video Question Answeringcited by 0
  11. 2026Knowledge Graph Enhanced Memory-Augmented Retrieval for Long Context Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Retrieval meets Long Context Large Language Models
  2. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation
  3. 2024Enhancing Long Context Performance in LLMs Through Inner Loop Query Mechanism
  4. 2026TA-Mem: Tool-Augmented Autonomous Memory Retrieval for LLM in Long-Term Conversational QA
  5. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
  6. 2024Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Survey
  7. 2025Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models
  8. 2023Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval