mlsanityData to 7 Oct 2026

Long-Context Language Modeling with Parallel Context Encoding

arXiv 2402.16617 ↗First posted 26 Feb 2024cs.CLcited by 84

Not on the map. Why?

Abstract (arXiv)

Extending large language models (LLMs) to process longer inputs is crucial for a wide range of applications. However, the substantial computational cost of transformers and limited generalization of positional encoding restrict the size of their context window. We introduce Context Expansion with Parallel Encoding (CEPE), a framework that can be applied to any existing decoder-only LLMs to extend their context window. CEPE employs a small encoder to process long inputs chunk by chunk, enabling the frozen decoder to utilize additional contexts via cross-attention. CEPE is efficient, generalizable, and versatile: trained with 8K-token documents, it extends the context window of LLAMA-2 to 128K tokens, offering 10x the throughput with only 1/6 of the memory. CEPE yields strong performance on language modeling and in-context learning. CEPE also excels in retrieval-augmented applications, while existing long-context models degenerate with retrieved contexts. We further introduce a CEPE variant that can extend the context window of instruction-tuned models using only unlabeled data, and showcase its effectiveness on LLAMA-2-CHAT, leading to a strong instruction-following model that can leverage very long contexts on downstream tasks.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 58 of its 78 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

58 of 78 references.

References · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
38 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  5. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  10. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  11. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  13. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  14. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  15. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  16. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  17. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  18. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  19. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  20. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  21. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  22. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  23. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  24. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  25. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  26. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  27. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  28. 2023Adapting Language Models to Compress Contextscited by 350
  29. 2021Efficient Attentions for Long Document Summarizationcited by 342
  30. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  31. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  32. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  33. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  34. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  35. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
  36. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  37. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  38. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 35; 2026: 18.

84 papers.

Sort:

By area:

  1. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  2. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  4. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  5. 2024InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationalescited by 100
  6. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  7. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  8. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  9. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  10. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  11. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35
  12. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 35
38 more
  1. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  2. 2024Large Language Models Can Self-Improve in Long-context Reasoningcited by 29
  3. 2024CodeS: Natural Language to Code Repository via Multi-Layer Sketchcited by 28
  4. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  5. 2024Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Modelscited by 26
  6. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answeringcited by 25
  7. 2025Vision-centric Token Compression in Large Language Modelcited by 21
  8. 2025PISCO: Pretty Simple Compression for Retrieval-Augmented Generationcited by 20
  9. 2024Entropy-Based Decoding for Retrieval-Augmented Large Language Modelscited by 19
  10. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 19
  11. 2024LLoCO: Learning Long Contexts Offlinecited by 16
  12. 2024KBLaM: Knowledge Base augmented Language Modelcited by 16
  13. 2026Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systemscited by 16
  14. 2024Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Modelcited by 15
  15. 2024E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoningcited by 13
  16. 2024On the token distance modeling ability of higher RoPE attention dimensioncited by 13
  17. 2024XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inferencecited by 12
  18. 2024Implicit In-context Learningcited by 11
  19. 2024MemLong: Memory-Augmented Retrieval for Long Text Modelingcited by 11
  20. 2025Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understandingcited by 11
  21. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  22. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  23. 2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 9
  24. 2025See the Text: From Tokenization to Visual Readingcited by 9
  25. 2024Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modelingcited by 8
  26. 2025AbsenceBench: Language Models Can't Tell What's Missingcited by 8
  27. 2025REFRAG: Rethinking RAG based Decodingcited by 8
  28. 2024Self-Calibrated Listwise Reranking with Large Language Modelscited by 7
  29. 2025Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Accesscited by 7
  30. 2024Squid: Long Context as a New Modality for Energy-Efficient On-Device Language Modelscited by 6
  31. 2025Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Modelscited by 6
  32. 2025Efficient Context Scaling with LongCat ZigZag Attentioncited by 6
  33. 2025Curse of High Dimensionality Issue in Transformer for Long-context Modelingcited by 5
  34. 2025ImpRAG: Retrieval-Augmented Generation with Implicit Queriescited by 5
  35. 2025End-to-End Long Document Summarization using Gradient Cachingcited by 4
  36. 2025CAFE: Retrieval Head-based Coarse-to-Fine Information Seeking to Enhance Multi-Document QA Capabilitycited by 4
  37. 2026End-to-End Context Compression at Scalecited by 4
  38. 2025BEE-RAG: Balanced Entropy Engineering for Retrieval-Augmented Generationcited by 3

50 most cited of 84 All 84 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  2. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
  3. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts
  4. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  5. 2023Retrieval meets Long Context Large Language Models
  6. 2024Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization
  7. 2024Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression
  8. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Models