mlsanityData to 14 Oct 2026

In-context Autoencoder for Context Compression in a Large Language Model

arXiv 2307.06945 ↗First posted 13 Jul 2023cs.CL cs.AI cs.LGcited by 199

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

We propose the In-context Autoencoder (ICAE), leveraging the power of a large language model (LLM) to compress a long context into short compact memory slots that can be directly conditioned on by the LLM for various purposes. ICAE is first pretrained using both autoencoding and language modeling objectives on massive text data, enabling it to generate memory slots that accurately and comprehensively represent the original context. Then, it is fine-tuned on instruction data for producing desirable responses to various prompts. Experiments demonstrate that our lightweight ICAE, introducing about 1% additional parameters, effectively achieves 4× context compression based on Llama, offering advantages in both improved latency and GPU memory cost during inference, and showing an interesting insight in memorization as well as potential for scalability. These promising results imply a novel perspective on the connection between working memory in cognitive science and representation learning in LLMs, revealing ICAE's significant implications in addressing the long context problem and suggesting further research in LLM context management. Our data, code and models are available at https://github.com/getao/icae.

Key facts. Cited by 199 papers in the map (data through 14 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 53 references that are in the map. First posted on arXiv 13 Jul 2023.

Builds on

34 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models6 Oct

Other references · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  11. 2020Longformer: The Long-Document Transformercited by 3,413
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
21 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  3. 2020Rethinking Attention with Performerscited by 1,601
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  5. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  7. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  8. 2022High Fidelity Neural Audio Compressioncited by 687
  9. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  10. 2023Adapting Language Models to Compress Contextscited by 356
  11. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  12. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 251
  13. 2022Memorizing Transformerscited by 226
  14. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  15. 2022Recurrent Memory Transformercited by 203
  16. 2022Learning by Distilling Contextcited by 138
  17. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  18. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 98
  19. 2022Linear Complexity Randomized Self-attention Mechanismcited by 31
  20. 2023Learn to Memorize: Scalable Continual Learning in Semiparametric Models with Mixture-of-Neighbors Induction Memorycited by 12
  21. 2022Extensible Prompts for Language Models on Zero-shot Language Style Customizationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 45; 2025: 51; 2026: 92.

199 papers.

Sort:

By area:

  1. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  2. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  3. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 364
  4. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  5. 2023Language Modeling Is Compressioncited by 225
  6. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  7. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  8. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  9. 2023Efficient Large Language Models: A Surveycited by 186
  10. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  12. 2024xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Tokencited by 125
38 more
  1. 2024mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understandingcited by 114
  2. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  3. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 97
  4. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  5. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
  6. 2024VoCo-LLaMA: Towards Vision Compression with Large Language Modelscited by 79
  7. 2024User-LLM: Efficient LLM Contextualization with User Embeddingscited by 76
  8. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergycited by 71
  9. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devicescited by 69
  10. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarioscited by 59
  11. 2024GOFA: A Generative One-For-All Model for Joint Graph Language Modelingcited by 57
  12. 2025Mem-α: Learning Memory Construction via Reinforcement Learningcited by 55
  13. 2025M+: Extending MemoryLLM with Scalable Long-Term Memorycited by 54
  14. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 50
  15. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  16. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 49
  17. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 48
  18. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 47
  19. 2023Compressed Context Memory For Online Language Model Interactioncited by 43
  20. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 38
  21. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  22. 2023Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruningcited by 26
  23. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answeringcited by 26
  24. 2024Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Passcited by 26
  25. 2025Dynamic Parametric Retrieval Augmented Generation for Test-time Knowledge Enhancementcited by 25
  26. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  27. 2026SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Passcited by 25
  28. 2024CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memorycited by 24
  29. 2025Training Plug-n-Play Knowledge Modules with Deep Context Distillationcited by 24
  30. 2026AgentOCR: Reimagining Agent History via Optical Self-Compressioncited by 22
  31. 2024Self-Updatable Large Language Models by Integrating Context into Model Parameterscited by 21
  32. 2025Vision-centric Token Compression in Large Language Modelcited by 21
  33. 2025GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignmentcited by 21
  34. 2025PISCO: Pretty Simple Compression for Retrieval-Augmented Generationcited by 20
  35. 2024Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Modelscited by 19
  36. 2024Characterizing Prompt Compression Methods for Long Context Inferencecited by 18
  37. 2024Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learningcited by 17
  38. 2025Context Cascade Compression: Exploring the Upper Limits of Text Compressioncited by 17

50 most cited of 199 All 199 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
  2. 2024In-Context Former: Lightning-fast Compressing Context for Large Language Model
  3. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  4. 2026Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
  5. 2024Prompt Compression for Large Language Models: A Survey
  6. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
  7. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  8. 2024Can Large Language Models Understand Context?