mlsanityData to 7 Oct 2026

LLoCO: Learning Long Contexts Offline

arXiv 2404.07979 ↗First posted 11 Apr 2024cs.CL cs.AI cs.LGcited by 16

On the map: Efficient LLM inference and serving → Prompt and context compression

Abstract (arXiv)

Processing long contexts remains a challenge for large language models (LLMs) due to the quadratic computational and memory overhead of the self-attention mechanism and the substantial KV cache sizes during generation. We propose LLoCO, a novel approach to address this problem by learning contexts offline through context compression and in-domain parameter-efficient finetuning with LoRA. Our method enables an LLM to create a concise representation of the original context and efficiently retrieve relevant information to answer questions accurately. Our approach extends the effective context window of a 4k token LLaMA2-7B model to handle up to 128k tokens. We evaluate our approach on several long-context question-answering datasets, demonstrating that LLoCO significantly outperforms in-context learning while using 30× fewer tokens during inference. LLoCO achieves up to 7.62× speed-up during inference and 11.52× higher throughput during finetuning, substantially reduces the cost of long document question answering. This makes it a promising solution for efficient long context processing. Our code is publicly available on https://github.com/jeffreysijuntan/lloco.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Prompt and context compression, one of 12 directions in Efficient LLM inference and serving. Builds on 44 of its 54 references that are in the map. First posted on arXiv 11 Apr 2024.

Builds on

44 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression19 Mar
  2. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression10 Oct
  3. 2023In-context Autoencoder for Context Compression in a Large Language Model13 Jul

Other references · 41

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
29 more
  1. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  2. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  3. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  5. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  6. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  8. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  9. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  12. 2023Active Retrieval Augmented Generationcited by 506
  13. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  14. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  15. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  16. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  17. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  18. 2023Adapting Language Models to Compress Contextscited by 350
  19. 2021Efficient Attentions for Long Document Summarizationcited by 342
  20. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  21. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  22. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  23. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  24. 2023Text Embeddings Reveal (Almost) As Much As Textcited by 189
  25. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  26. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 146
  27. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  28. 2024Learning to Route Among Specialized Experts for Zero-Shot Generalizationcited by 65
  29. 2023TOAST: Transfer Learning via Attention Steeringcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 7.

16 papers.

Sort:

By area:

  1. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  2. 2025A Survey of Research in Large Language Models for Electronic Design Automationcited by 38
  3. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  4. 2025Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Taskscited by 27
  5. 2024Context Embeddings for Efficient Answer Generation in RAGcited by 26
  6. 2024An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encodingcited by 16
  7. 2024E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoningcited by 13
  8. 2026COMI: Coarse-to-fine Context Compression via Marginal Information Gaincited by 13
  9. 2024Squid: Long Context as a New Modality for Energy-Efficient On-Device Language Modelscited by 6
  10. 2026End-to-End Context Compression at Scalecited by 4
  11. 2026Beyond Position Bias: Shifting Context Compression from Position-Driven to Semantic-Drivencited by 3
  12. 2026Language Models Need Sleep: Learning to Self-Modify and Consolidate Memoriescited by 1
4 more
  1. 2026CoMem: Reusing Transformer Depth across Queries with Persistent Intermediate Residualscited by 1
  2. 2025Context-Aware Hierarchical Learning: A Two-Step Paradigm towards Safer LLMscited by 0
  3. 2026Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAGcited by 0
  4. 2026DimPO: Dimensionality Reduction for Attention using Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  2. 2026Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
  3. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA
  4. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  5. 2024QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compression
  6. 2024Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference
  7. 2024Large Language Models Can Self-Improve in Long-context Reasoning
  8. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answering