mlsanityData to 7 Oct 2026

Training-Free Long-Context Scaling of Large Language Models

arXiv 2402.17463 ↗First posted 27 Feb 2024cs.CLcited by 61

Not on the map. Why?

Abstract (arXiv)

The ability of Large Language Models (LLMs) to process and generate coherent text is markedly weakened when the number of input tokens exceeds their pretraining length. Given the expensive overhead of finetuning large-scale models with longer sequences, we propose Dual Chunk Attention (DCA), which enables Llama2 70B to support context windows of more than 100k tokens without continual training. By decomposing the attention computation for long sequences into chunk-based modules, DCA manages to effectively capture the relative positional information of tokens within the same chunk (Intra-Chunk) and across distinct chunks (Inter-Chunk), as well as integrates seamlessly with Flash Attention. In addition to its impressive extrapolation capability, DCA achieves performance on practical long-context tasks that is comparable to or even better than that of finetuned models. When compared with proprietary models, our training-free 70B model attains 94% of the performance of gpt-3.5-16k, indicating it is a viable open-source alternative. All code and data used in this work are released at urlhttps://github.com/HKUNLP/ChunkLlama.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

31 of 48 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  8. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  9. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  10. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  11. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  12. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
19 more
  1. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  2. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  3. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  4. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  5. 2023Compositional Exemplars for In-context Learningcited by 143
  6. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  7. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  8. 2024Long Context Compression with Activation Beaconcited by 101
  9. 2022Parallel Context Windows for Large Language Modelscited by 81
  10. 2023Prompted LLMs as Chatbot Modules for Long Open-domain Conversationcited by 61
  11. 2023Learning to Retrieve In-Context Examples for Large Language Modelscited by 60
  12. 2024Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Modelscited by 57
  13. 2023Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Datacited by 48
  14. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 43
  15. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 39
  16. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 23
  17. 2023Monotonic Location Attention for Length Generalizationcited by 9
  18. 2023Zebra: Extending Context Window with Layerwise Grouped Local-Global Attentioncited by 6
  19. 2023Linear Attention via Orthogonal Memorycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 24; 2026: 13.

61 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2024Qwen2 Technical Reportcited by 2,076
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  6. 2025Qwen2.5-1M Technical Reportcited by 158
  7. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  8. 2024LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMscited by 104
  9. 2024Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scalecited by 87
  10. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 60
  11. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  12. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 38
38 more
  1. 2024Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformerscited by 32
  2. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densitiescited by 32
  3. 2024MambaLRP: Explaining Selective State Space Sequence Modelscited by 31
  4. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  5. 2025Cognitive Memory in Large Language Modelscited by 28
  6. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  7. 2024HashAttention: Semantic Sparsity for Faster Inferencecited by 23
  8. 2025LongRoPE2: Near-Lossless LLM Context Window Scalingcited by 21
  9. 2025A Survey on LLM Mid-Trainingcited by 18
  10. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 16
  11. 20243D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encodingcited by 13
  12. 2024ReAttention: Training-Free Infinite Context with Finite Attention Scopecited by 13
  13. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolationcited by 13
  14. 2026LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewardscited by 13
  15. 2025Implicit Search via Discrete Diffusion: A Study on Chesscited by 11
  16. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  17. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  18. 2025LongAttn: Selecting Long-context Training Data via Token-level Attentioncited by 9
  19. 2025Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Accesscited by 7
  20. 2024ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Modelscited by 6
  21. 2025Curse of High Dimensionality Issue in Transformer for Long-context Modelingcited by 5
  22. 2025SuperWriter: Reflection-Driven Long-Form Generation with Large Language Modelscited by 5
  23. 2025PiKV: KV Cache Management System for Mixture of Expertscited by 5
  24. 2024Automatically Generating Numerous Context-Driven SFT Data for LLMs across Diverse Granularitycited by 4
  25. 2024Longer is (Not Necessarily) Stronger: Punctuated Long-Sequence Training for Enhanced Speech Recognition and Translationcited by 4
  26. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  27. 2025PERK: Long-Context Reasoning as Test-Time Learningcited by 4
  28. 2026Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contextscited by 4
  29. 2026HyMem: Hybrid Memory Architecture with Dynamic Retrieval Schedulingcited by 4
  30. 2025ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Modelscited by 3
  31. 2026Learning When to Attend: Conditional Memory Access for Long-Context LLMscited by 3
  32. 2026The Y-Combinator for LLMs: Solving Long-Context Rot with λ-Calculuscited by 3
  33. 2024GIRAFFE: Design Choices for Extending the Context Length of Visual Language Modelscited by 2
  34. 2025Enhancing Complex Instruction Following for Large Language Models with Mixture-of-Contexts Fine-tuningcited by 2
  35. 2026HiCI: Hierarchical Construction-Integration for Long-Context Attentioncited by 2
  36. 2026A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraintscited by 2
  37. 2024MammothModa: Multi-Modal Large Language Modelcited by 1
  38. 2024Two are better than one: Context window extension with multi-grained self-injectioncited by 1

50 most cited of 61 All 61 (JSON)