mlsanityData to 14 Oct 2026

dots.llm1 Technical Report

arXiv 2506.05767 ↗First posted 6 Jun 2025cs.CL cs.AIcited by 5

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Mixture of Experts (MoE) models have emerged as a promising paradigm for scaling language models efficiently by activating only a subset of parameters for each input token. In this report, we present dots.llm1, a large-scale MoE model that activates 14B parameters out of a total of 142B parameters, delivering performance on par with state-of-the-art models while reducing training and inference costs. Leveraging our meticulously crafted and efficient data processing pipeline, dots.llm1 achieves performance comparable to Qwen2.5-72B after pretraining on 11.2T high-quality tokens and post-training to fully unlock its capabilities. Notably, no synthetic data is used during pretraining. To foster further research, we open-source intermediate training checkpoints at every one trillion tokens, providing valuable insights into the learning dynamics of large language models.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 55 of its 78 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

55 of 78 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Qwen2.5 Technical Report19 Dec
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model7 May
  3. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models11 Jan
  4. 2024Mixtral of Experts8 Jan

Other references · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  7. 2020GLU Variants Improve Transformercited by 1,804
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  13. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  14. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  15. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  16. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  17. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  18. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  19. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  20. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  21. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  22. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  23. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  24. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  25. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  26. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  27. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  28. 20242 OLMo 2 Furiouscited by 340
  29. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 287
  30. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  31. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  32. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  33. 2019Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensioncited by 81
  34. 2024Evaluating Language Models for Efficient Code Generationcited by 73
  35. 2024Are Your LLMs Capable of Stable Reasoning?cited by 65
  36. 2024Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Modelscited by 43
  37. 2024Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Modelscited by 9
  38. 2024Style Mixture of Experts for Expressive Text-To-Speech Synthesiscited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025UltraMemV2: Memory Networks Scaling to 120B Parameters with Superior Long-Context Learningcited by 5
  2. 2025EntropyLong: Effective Long-Context Training via Predictive Uncertaintycited by 5
  3. 2025LiteLong: Resource-Efficient Long-Context Data Synthesis for LLMscited by 3
  4. 2025RedOne 2.0: Rethinking Domain-specific LLM Post-Training in Social Networking Servicescited by 1
  5. 2026CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Scaling Fine-Grained MoE Beyond 50B Parameters: Empirical Evaluation and Practical Insights
  2. 2025Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
  3. 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
  4. 2024A Survey on Mixture of Experts in Large Language Models
  5. 2024Toward Inference-optimal Mixture-of-Expert Large Language Models
  6. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
  7. 2024Scaling Laws for Fine-Grained Mixture of Experts
  8. 2025Faster MoE LLM Inference for Extremely Large Models