mlsanityData to 14 Oct 2026

Qwen2.5-1M Technical Report

arXiv 2501.15383 ↗First posted 26 Jan 2025cs.CLcited by 164

Not on the map. Why?

Cited by 44 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Qwen2.5-1M, a series of models that extend the context length to 1 million tokens. Compared to the previous 128K version, the Qwen2.5-1M series have significantly enhanced long-context capabilities through long-context pre-training and post-training. Key techniques such as long data synthesis, progressive pre-training, and multi-stage supervised fine-tuning are employed to effectively enhance long-context performance while reducing training costs. To promote the use of long-context models among a broader user base, we present and open-source our inference framework. This framework includes a length extrapolation method that can expand the model context lengths by at least four times, or even more, without additional training. To reduce inference costs, we implement a sparse attention method along with chunked prefill optimization for deployment scenarios and a sparsity refinement method to improve precision. Additionally, we detail our optimizations in the inference engine, including kernel optimization, pipeline parallelism, and scheduling optimization, which significantly enhance overall inference performance. By leveraging our inference framework, the Qwen2.5-1M models achieve a remarkable 3x to 7x prefill speedup in scenarios with 1 million tokens of context. This framework provides an efficient and powerful solution for developing applications that require long-context processing using open-source models. The Qwen2.5-1M series currently includes the open-source models Qwen2.5-7B-Instruct-1M and Qwen2.5-14B-Instruct-1M, as well as the API-accessed model Qwen2.5-Turbo. Evaluations show that Qwen2.5-1M models have been greatly improved in long-context tasks without compromising performance in short-context scenarios. Specifically, the Qwen2.5-14B-Instruct-1M model significantly outperforms GPT-4o-mini in long-context tasks and supports contexts eight times longer.

Key facts. Cited by 164 papers in the map (data through 14 Oct 2026). Builds on 36 of its 46 references that are in the map. First posted on arXiv 26 Jan 2025.

Builds on

36 of 46 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2024Qwen2.5 Technical Reportcited by 4,942
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
24 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,637
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  7. 2024Mixtral of Expertscited by 1,823
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  11. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  12. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  13. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  14. 2016Language Modeling with Gated Convolutional Networkscited by 702
  15. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  16. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  17. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  18. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  19. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  20. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 106
  21. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 81
  22. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 63
  23. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  24. 2023Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformerscited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 83; 2026: 81.

164 papers.

Sort:

By area:

  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  2. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 246
  3. 2025Qwen3Guard Technical Reportcited by 155
  4. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 98
  5. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 95
  6. 2026AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applicationscited by 53
  7. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
  8. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 46
  9. 2025Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agentscited by 38
  10. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learningcited by 34
  11. 2025Semantic-Aware Logical Reasoning via a Semiotic Frameworkcited by 32
  12. 2026OPSDL: On-Policy Self-Distillation for Long-Context Language Modelscited by 30
38 more
  1. 2026Logical Phase Transitions: Understanding Collapse in LLM Logical Reasoningcited by 25
  2. 2025UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learningcited by 24
  3. 2025Mid-Training of Large Language Models: A Surveycited by 21
  4. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 19
  5. 2026Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoningcited by 19
  6. 2025ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Modelscited by 17
  7. 2025CacheClip: Accelerating RAG with Effective KV Cache Reusecited by 16
  8. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Evictioncited by 16
  9. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  10. 2026MemOCR: Layout-Aware Visual Memory for Efficient Long-Horizon Reasoningcited by 15
  11. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolationcited by 14
  12. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Modelscited by 14
  13. 2025Mixed-R1: Unified Reward Perspective For Reasoning Capability in Multimodal Large Language Modelscited by 13
  14. 2026MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Modelscited by 12
  15. 2025MAPO: Mixed Advantage Policy Optimizationcited by 11
  16. 2025NP-Engine: Empowering Optimization Reasoning in Large Language Models with Verifiable Synthetic NP Problemscited by 11
  17. 2025Rectified Sparse Attentioncited by 10
  18. 2025Fidel-TS: A High-Fidelity Multimodal Benchmark for Time Series Forecastingcited by 10
  19. 2026Apollo: Unified Multi-Task Audio-Video Joint Generationcited by 10
  20. 2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agentscited by 10
  21. 2026REAM: Merging Improves Pruning of Experts in LLMscited by 10
  22. 2025Cascading Adversarial Bias from Injection to Distillation in Language Modelscited by 9
  23. 2025VER-Bench: Evaluating MLLMs on Reasoning with Fine-Grained Visual Evidencecited by 9
  24. 2025ProxyAttn: Guided Sparse Attention via Representative Headscited by 9
  25. 2026AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rolloutscited by 9
  26. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 9
  27. 2026FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefillingcited by 9
  28. 2026Towards Real-world Human Behavior Simulation: Benchmarking Large Language Models on Long-horizon, Cross-scenario, Heterogeneous Behavior Tracescited by 9
  29. 2025Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?cited by 8
  30. 2025Towards Multi-Agent Reasoning Systems for Collaborative Expertise Delegation: An Exploratory Design Studycited by 8
  31. 2025Reasoning BO: Enhancing Bayesian Optimization with Long-Context Reasoning Power of LLMscited by 8
  32. 2025SATURN: SAT-based Reinforcement Learning to Unleash LLMs Reasoningcited by 8
  33. 2025AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animationcited by 8
  34. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  35. 2025LooGLE v2: Are LLMs Ready for Real World Long Dependency Challenges?cited by 8
  36. 2026Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learningcited by 8
  37. 2026Scheduling LLM Inference with Uncertainty-Aware Output Length Predictionscited by 8
  38. 2025Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoningcited by 7

50 most cited of 164 All 164 (JSON)