mlsanityData to 7 Oct 2026

Effective Long-Context Scaling of Foundation Models

arXiv 2309.16039 ↗First posted 27 Sep 2023cs.CLcited by 301

Not on the map. Why?

Cited by 28 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our model series are built through continual pretraining from Llama 2 with longer training sequences and on a dataset where long texts are upsampled. We perform extensive evaluation on language modeling, synthetic context probing tasks, and a wide range of research benchmarks. On research benchmarks, our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2. Notably, with a cost-effective instruction tuning procedure that does not require human-annotated long instruction data, the 70B variant can already surpass gpt-3.5-turbo-16k's overall performance on a suite of long-context tasks. Alongside these results, we provide an in-depth analysis on the individual components of our method. We delve into Llama's position encodings and discuss its limitation in modeling long dependencies. We also examine the impact of various design choices in the pretraining process, including the data mix and the training curriculum of sequence lengths -- our ablation experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences.

Key facts. Cited by 301 papers in the map (data through 7 Oct 2026). Builds on 48 of its 66 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

48 of 66 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
36 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  13. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  14. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  15. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  16. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  17. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  18. 2018Self-Attention with Relative Position Representationscited by 1,112
  19. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  20. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  21. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  22. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  23. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  24. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  25. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  26. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  27. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  28. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  29. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  30. 2021Random Feature Attentioncited by 350
  31. 2020Sparse Sinkhorn Attentioncited by 250
  32. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  33. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  34. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  35. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  36. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 22; 2024: 129; 2025: 105; 2026: 45.

301 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  4. 2024Qwen2 Technical Reportcited by 2,076
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  6. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  7. 2024Yi: Open Foundation Models by 01.AIcited by 706
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  9. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  10. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  11. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  12. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 267
38 more
  1. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  2. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  3. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  4. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  5. 2024Many-Shot In-Context Learningcited by 203
  6. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  7. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  8. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  9. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  10. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  11. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  12. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  13. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  14. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 168
  15. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  16. 2025Qwen2.5-1M Technical Reportcited by 158
  17. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  18. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  19. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  20. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  21. 2023OpenAgents: An Open Platform for Language Agents in the Wildcited by 136
  22. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  23. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  24. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 123
  25. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 121
  26. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  27. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 120
  28. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 118
  29. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  30. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  31. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  32. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
  33. 2025VSA: Faster Video Diffusion with Trainable Sparse Attentioncited by 100
  34. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 97
  35. 2024MEMORYLLM: Towards Self-Updatable Large Language Modelscited by 89
  36. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 89
  37. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 87
  38. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84

50 most cited of 301 All 301 (JSON)