mlsanityData to 13 Oct 2026

Compressed Context Memory For Online Language Model Interaction

arXiv 2312.03414 ↗First posted 6 Dec 2023cs.LG cs.CLcited by 42

Not on the map. Why?

Cited by 22 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents a context key/value compression method for Transformer language models in online scenarios, where the context continually expands. As the context lengthens, the attention process demands increasing memory and computations, which in turn reduces the throughput of the language model. To address this challenge, we propose a compressed context memory system that continually compresses the accumulating attention key/value pairs into a compact memory space, facilitating language model inference in a limited memory space of computing environments. Our compression process involves integrating a lightweight conditional LoRA into the language model's forward pass during inference, without the need for fine-tuning the model's entire set of weights. We achieve efficient training by modeling the recursive compression process as a single parallelized forward computation. Through evaluations on conversation, personalization, and multi-task learning, we demonstrate that our approach achieves the performance level of a full context model with 5× smaller context memory size. We further demonstrate the applicability of our approach in a streaming setting with an unlimited context length, outperforming the sliding window approach. Codes are available at https://github.com/snu-mllab/context-memory.

Key facts. Cited by 42 papers in the map (data through 13 Oct 2026). Builds on 37 of its 41 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

37 of 41 references.

References · 37

  1. 2017Attention Is All You Needcited by 60,106
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,007
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,379
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,137
  9. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,994
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,835
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,681
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,538
25 more
  1. 2015Deep Learning and the Information Bottleneck Principlecited by 796
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 707
  3. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  4. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 551
  5. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 458
  6. 2021MetaICL: Learning to Learn In Contextcited by 452
  7. 2023LaMP: When Large Language Models Meet Personalizationcited by 430
  8. 2023Learning to Compress Prompts with Gist Tokenscited by 382
  9. 2014Memory Networkscited by 360
  10. 2023Adapting Language Models to Compress Contextscited by 351
  11. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  12. 2022Memorizing Transformerscited by 222
  13. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 209
  14. 2022Recurrent Memory Transformercited by 199
  15. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 194
  16. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  17. 2016Using Fast Weights to Attend to the Recent Pastcited by 164
  18. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  19. 2022Learning by Distilling Contextcited by 132
  20. 2022Block-Recurrent Transformerscited by 110
  21. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 96
  22. 2016Memory-Efficient Backpropagation Through Timecited by 78
  23. 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 45
  24. 2023Meta-Learning Online Adaptation of Language Modelscited by 43
  25. 2022Meta-Learning Fast Weight Language Modelscited by 21

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 10; 2025: 17; 2026: 15.

42 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 126
  2. 2024Long Context Compression with Activation Beaconcited by 101
  3. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 91
  4. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 51
  5. 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
  6. 2025Cognitive Memory in Large Language Modelscited by 29
  7. 2024Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Passcited by 26
  8. 2024HMT: Hierarchical Memory Transformer for Efficient Long Context Language Processingcited by 20
  9. 2025InfiniteICL: Breaking the Limit of Context Window Size via Long Short-term Memory Transformationcited by 16
  10. 2026Latent Context Compilation: Distilling Long Context into Compact Portable Memorycited by 13
  11. 2025Long Context In-Context Compression by Getting to the Gist of Gistingcited by 12
  12. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 9
30 more
  1. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learningcited by 9
  2. 2025SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compressioncited by 9
  3. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attentioncited by 7
  4. 2025AdmTree: Compressing Lengthy Context with Adaptive Semantic Treescited by 7
  5. 2024CMT: A Memory Compression Method for Continual Knowledge Learning of Large Language Modelscited by 6
  6. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 6
  7. 2025CompLLM: Compression for Long Context Q&Acited by 5
  8. 2024Compressing Lengthy Context With UltraGistcited by 4
  9. 2024Training-Free Exponential Context Extension via Cascading KV Cachecited by 4
  10. 2026Pyramid Forcing: Head-Aware Pyramid KV Cache Policy for High-Quality Long Video Generationcited by 4
  11. 2025FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Frameworkcited by 3
  12. 2025AirCache: Activating Inter-modal Relevancy KV Cache Compression for Efficient Large Vision-Language Model Inferencecited by 3
  13. 2024EarthquakeNPP: A Benchmark for Earthquake Forecasting with Neural Point Processescited by 2
  14. 2025High-Layer Attention Pruning with Rescalingcited by 1
  15. 2025Learning What to Remember: Adaptive Probabilistic Memory Retention for Memory-Efficient Language Modelscited by 1
  16. 2026Mixture of Chapters: Scaling Learnt Memory in Transformerscited by 1
  17. 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1
  18. 2026Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inferencecited by 1
  19. 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstructioncited by 1
  20. 2025Dynamic Long Short-Term Memory Based Memory Storage For Long Horizon LLM Interactioncited by 0
  21. 2025Rhea: Role-aware Heuristic Episodic Attention for Conversational LLMscited by 0
  22. 2026Multi-Head Low-Rank Attentioncited by 0
  23. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategiescited by 0
  24. 2026Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Servingcited by 0
  25. 2026Out of Sight: Compression-Aware Content Protection against Agentic Crawlerscited by 0
  26. 2026Adapting Tree-Structured Speculative Decoding to DeepSeek-V4 for Efficient Inferencecited by 0
  27. 2026HasMem: Hard-Origin Adaptively Softened Memory for Long-Term LLM Agentscited by 0
  28. 2026DimPO: Dimensionality Reduction for Attention using Preference Optimizationcited by 0
  29. 2026ReLMem: Learning Recurrent Memory for Longitudinal EHR Modelingcited by 0
  30. 2026Role-aware Heuristic Episodic Attention for Conversational LLMscited by 0