Data to 7 Oct 2026

Memformer: A Memory-Augmented Transformer for Sequence Modeling

arXiv 2010.06891 ↗First posted 14 Oct 2020cs.CLcited by 63

Not on the map. Why?

Abstract (arXiv)

Transformers have reached remarkable success in sequence modeling. However, these models have efficiency issues as they need to store all the history token-level representations as memory. We present Memformer, an efficient neural network for sequence modeling, that utilizes an external dynamic memory to encode and retrieve past information. Our model achieves linear time complexity and constant memory space complexity when processing long sequences. We also propose a new optimization scheme, memory replay back-propagation (MRBP), which promotes long-range back-propagation through time with a significantly reduced memory requirement. Experimental results show that Memformer has achieved comparable performance compared to the baselines by using 8.1x less memory space and 3.2x faster on inference. Analysis of the attention pattern shows that our external memory slots can encode and retain important information through timesteps.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 27; 2025: 13; 2026: 8.

63 papers.

Sort:

By area:

2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
2024Titans: Learning to Memorize at Test Timecited by 303
2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
2023Efficient Large Language Models: A Surveycited by 183
2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
2024MEMORYLLM: Towards Self-Updatable Large Language Modelscited by 89
2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71

38 more

2023Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Modelscited by 62
2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
2024Compress to Impress: Unleashing the Potential of Compressive Memory in Real-World Long-Term Conversationscited by 36
2024MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memorycited by 35
2024Foundation Models for Music: A Surveycited by 34
2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 31
2024HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processingcited by 20
2026Evaluating Memory Structure in LLM Agentscited by 12
2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
2024MambaEVT: Event Stream based Visual Object Tracking using State Space Modelcited by 11
2025Episodic memory in AI agents poses risks that should be studied and mitigatedcited by 11
2023From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformerscited by 10
2023Memory Efficient Neural Processes via Constant Memory Attention Blockcited by 9
2025Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contextscited by 9
2025Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architecturescited by 9
2025MeSH: Memory-as-State-Highways for Recursive Transformerscited by 8
2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6
2024Investigating Recurrent Transformers with Dynamic Haltcited by 5
2026CoMeT: Collaborative Memory Transformer for Efficient Long Context Modelingcited by 5
2026Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoningcited by 5
2026Pretraining Recurrent Networks without Recurrencecited by 5
2023Diable: Efficient Dialogue State Tracking as Operations on Tablescited by 4
2024Multi-Modal Retrieval For Large Language Model Based Speech Recognitioncited by 4
2024Attendre: Wait To Attend By Retrieval With Evicted Queries in Memory-Based Transformers for Long Context Processingcited by 3
2024The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Servingcited by 3
2025DAM-Seg: Anatomically accurate cardiac segmentation using Dense Associative Networkscited by 3
2025Vision encoders should be image size agnostic and task drivencited by 3
2023Improving Autoregressive NLP Tasks via Modular Linearized Attentioncited by 2
2023A Memory-Augmented Multi-Task Collaborative Framework for Unsupervised Traffic Accident Detection in Driving Videoscited by 2
2023A Framework for Inference Inspired by Human Memory Mechanismscited by 2
2023Uncertainty Guided Global Memory Improves Multi-Hop Question Answeringcited by 2
2024Sound Source Separation Using Latent Variational Block-Wise Disentanglementcited by 2
2024LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportionscited by 2
2024Token Turing Machines are Efficient Vision Modelscited by 2
2025LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMscited by 2
2023HIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory Transformercited by 1
2024You Only Use Reactive Attention Slice For Long Context Retrievalcited by 1

50 most cited of 63 All 63 (JSON)