mlsanityData to 14 Oct 2026

Parallel Context Windows for Large Language Models

arXiv 2212.10947 ↗First posted 21 Dec 2022cs.CLcited by 81

Not on the map. Why?

Abstract (arXiv)

When applied to processing long text, Large Language Models (LLMs) are limited by their context window. Existing efforts to address this limitation involve training specialized architectures, and cannot be easily applied to off-the-shelf LLMs. We present Parallel Context Windows (PCW), a method that alleviates the context window restriction for any off-the-shelf LLM without further training. The key to the approach is to carve a long context into chunks (``windows''), restrict the attention mechanism to apply only within each window, and re-use the positional embeddings across the windows. Our main results test the PCW approach on in-context learning with models that range in size between 750 million and 178 billion parameters, and show substantial improvements for tasks with diverse input and output spaces. We show additional benefits in other settings where long context windows may be beneficial: multi-hop questions and retrieval-augmented question answering with multiple retrieved documents. Our results highlight Parallel Context Windows as a promising method for applying off-the-shelf LLMs in a range of settings that require long text sequences. We make our code publicly available at https://github.com/ai21labs/parallel-context-windows.

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). Builds on 27 of its 60 references that are in the map. First posted on arXiv 21 Dec 2022.

Builds on

27 of 60 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  10. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  11. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
15 more
  1. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  2. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  3. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  4. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  5. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
  6. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  7. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  8. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  9. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
  10. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  11. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  12. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  13. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  14. 2021RAFT: A Real-World Few-Shot Text Classification Benchmarkcited by 44
  15. 2022Standing on the Shoulders of Giant Frozen Language Modelscited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 35; 2025: 25; 2026: 13.

81 papers.

Sort:

By area:

  1. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  2. 2023A Comprehensive Overview of Large Language Modelscited by 648
  3. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  4. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  5. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  6. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  7. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 124
  8. 2023Retrieval meets Long Context Large Language Modelscited by 115
  9. 2024A Survey on Large Language Model-Based Game Agentscited by 112
  10. 2023The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversationcited by 109
  11. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 108
  12. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 101
38 more
  1. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  2. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 83
  3. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  4. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
  5. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  6. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 58
  7. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  8. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansioncited by 50
  9. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 48
  10. 2024Memory3: Language Modeling with Explicit Memorycited by 40
  11. 2024Block-Attention for Efficient Prefillingcited by 37
  12. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  13. 2024PDE Generalization of In-Context Operator Networks: A Study on 1D Scalar Nonlinear Conservation Lawscited by 31
  14. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  15. 2024Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selectioncited by 25
  16. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 25
  17. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 24
  18. 2024Private prediction for large-scale synthetic text generationcited by 24
  19. 2025Vision-centric Token Compression in Large Language Modelcited by 21
  20. 2024Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learningcited by 20
  21. 2024KBLaM: Knowledge Base augmented Language Modelcited by 16
  22. 2024Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generationcited by 15
  23. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolationcited by 14
  24. 2024Differential Transformercited by 13
  25. 2025Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attentioncited by 12
  26. 2024TR2MTL: LLM based framework for Metric Temporal Logic Formalization of Traffic Rulescited by 9
  27. 2024XL3M: A Training-free Framework for LLM Length Extension Based on Segment-wise Inferencecited by 8
  28. 2025Rethinking Invariance in In-context Learningcited by 8
  29. 2024BLAZE: Cross-Language and Cross-Project Bug Localization via Dynamic Chunking and Hard Example Learningcited by 7
  30. 2024Memorization in In-Context Learningcited by 6
  31. 2025Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Modelscited by 6
  32. 2023Revisiting Parallel Context Windows: A Frustratingly Simple Alternative and Chain-of-Thought Deteriorationcited by 5
  33. 2024LAMPO: Large Language Models as Preference Machines for Few-shot Ordinal Classificationcited by 5
  34. 2025End-to-End Long Document Summarization using Gradient Cachingcited by 4
  35. 2025Schema for In-Context Learningcited by 4
  36. 2024MGIMM: Multi-Granularity Instruction Multimodal Model for Attribute-Guided Remote Sensing Image Detailed Descriptioncited by 3
  37. 2026Addressing Corpus Knowledge Poisoning Attacks on RAG Using Sparse Attentioncited by 3
  38. 2026SOCKET: SOft Collision Kernel EsTimator for Sparse Attentioncited by 3

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ACER: Automatic Language Model Context Extension via Retrieval
  2. 2024Two are better than one: Context window extension with multi-grained self-injection
  3. 2024Recurrent Context Compression: Efficiently Expanding the Context Window of LLM
  4. 2026Stacked from One: Multi-Scale Self-Injection for Context Window Extension
  5. 2025Infinite Retrieval: Attention Enhanced LLMs in Long-Context Processing
  6. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  7. 2025Beyond In-Context Learning: Aligning Long-form Generation of Large Language Models via Task-Inherent Attribute Guidelines
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding