mlsanityData to 14 Oct 2026

Qwen2.5 Technical Report

arXiv 2412.15115 ↗First posted 19 Dec 2024cs.CLcited by 4,942

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

In this report, we introduce Qwen2.5, a comprehensive series of large language models (LLMs) designed to meet diverse needs. Compared to previous iterations, Qwen 2.5 has been significantly improved during both the pre-training and post-training stages. In terms of pre-training, we have scaled the high-quality pre-training datasets from the previous 7 trillion tokens to 18 trillion tokens. This provides a strong foundation for common sense, expert knowledge, and reasoning capabilities. In terms of post-training, we implement intricate supervised finetuning with over 1 million samples, as well as multistage reinforcement learning. Post-training techniques enhance human preference, and notably improve long text generation, structural data analysis, and instruction following. To handle diverse and varied use cases effectively, we present Qwen2.5 LLM series in rich sizes. Open-weight offerings include base and instruction-tuned models, with quantized versions available. In addition, for hosted solutions, the proprietary models currently include two mixture-of-experts (MoE) variants: Qwen2.5-Turbo and Qwen2.5-Plus, both available from Alibaba Cloud Model Studio. Qwen2.5 has demonstrated top-tier performance on a wide range of benchmarks evaluating language understanding, reasoning, mathematics, coding, human preference alignment, etc. Specifically, the open-weight flagship Qwen2.5-72B-Instruct outperforms a number of open and proprietary models and demonstrates competitive performance to the state-of-the-art open-weight model, Llama-3-405B-Instruct, which is around 5 times larger. Qwen2.5-Turbo and Qwen2.5-Plus offer superior cost-effectiveness while performing competitively against GPT-4o-mini and GPT-4o respectively. Additionally, as the foundation, Qwen2.5 models have been instrumental in training specialized models such as Qwen2.5-Math, Qwen2.5-Coder, QwQ, and multimodal models.

Key facts. Cited by 4,942 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 57 of its 100 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

57 of 100 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  2. 2024Mixtral of Experts8 Jan
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun

Other references · 53

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2023Mistral 7Bcited by 3,164
38 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  3. 2024Qwen2 Technical Reportcited by 2,120
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  10. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  12. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  13. 2024Yi: Open Foundation Models by 01.AIcited by 719
  14. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  15. 2023The Falcon Series of Open Language Modelscited by 535
  16. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  17. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  18. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  19. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  20. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  21. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  22. 2024Are We Done with MMLU?cited by 194
  23. 2024BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languagescited by 173
  24. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  25. 2024Nemotron-4 340B Technical Reportcited by 144
  26. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
  27. 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignmentcited by 104
  28. 2024How to Evaluate Reward Models for RLHFcited by 88
  29. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 81
  30. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  31. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 62
  32. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 61
  33. 2024Towards Scalable Automated Alignment of LLMs: A Surveycited by 35
  34. 2023Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformerscited by 30
  35. 2024P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMscited by 29
  36. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
  37. 2024Language Models can Self-Lengthen to Generate Long Textscited by 24
  38. 2024MERA: A Comprehensive LLM Evaluation in Russiancited by 22

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 2798; 2026: 2141.

4,942 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  3. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  5. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  6. 2025Large Language Diffusion Modelscited by 912
  7. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  8. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  9. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 625
  10. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  11. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  12. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
38 more
  1. 2025Process Reinforcement through Implicit Rewardscited by 504
  2. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  3. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  4. 2025Muon is Scalable for LLM Trainingcited by 428
  5. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  6. 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 413
  7. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 410
  8. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 392
  9. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  10. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  11. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 335
  12. 2026Reinforcement Learning via Self-Distillationcited by 332
  13. 2024Titans: Learning to Memorize at Test Timecited by 328
  14. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  15. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  16. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 315
  17. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  18. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  19. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  20. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  21. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  22. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  23. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  24. 2025Learning to Reason under Off-Policy Guidancecited by 281
  25. 2025Reasoning with Exploration: An Entropy Perspectivecited by 281
  26. 2025TTRL: Test-Time Reinforcement Learningcited by 275
  27. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  28. 2025Seedance 1.0: Exploring the Boundaries of Video Generation Modelscited by 259
  29. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 255
  30. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  31. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 250
  32. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 246
  33. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 238
  34. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  35. 2025WebSailor: Navigating Super-human Reasoning for Web Agentcited by 229
  36. 2025MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understandingcited by 224
  37. 2025Kimi-Audio Technical Reportcited by 222
  38. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221

50 most cited of 4,942 All 4,942 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Technical Report: Full-Stack Fine-Tuning for the Q Programming Language
  2. 2023YAYI 2: Multilingual Open-Source Large Language Models
  3. 2025ShiQ: Bringing back Bellman to LLMs
  4. 2025Multilingual Language Model Pretraining using Machine-translated Data
  5. 2025Qwen3-VL Technical Report
  6. 2025Post-training Large Language Models for Diverse High-Quality Responses
  7. 2025Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resources
  8. 2025LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models