mlsanityData to 14 Oct 2026

YuLan-Mini: An Open Data-efficient Language Model

arXiv 2412.17743 ↗First posted 23 Dec 2024cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Effective pre-training of large language models (LLMs) has been challenging due to the immense resource demands and the complexity of the technical processes involved. This paper presents a detailed technical report on YuLan-Mini, a highly capable base model with 2.42B parameters that achieves top-tier performance among models of similar parameter scale. Our pre-training approach focuses on enhancing training efficacy through three key technical contributions: an elaborate data pipeline combines data cleaning with data schedule strategies, a robust optimization method to mitigate training instability, and an effective annealing approach that incorporates targeted data selection and long context training. Remarkably, YuLan-Mini, trained on 1.08T tokens, achieves performance comparable to industry-leading models that require significantly more data. To facilitate reproduction, we release the full details of the data composition for each training phase. Project details can be accessed at the following link: https://github.com/RUC-GSAI/YuLan-Mini.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 64 of its 135 references that are in the map. First posted on arXiv 23 Dec 2024.

Builds on

64 of 135 references.

References · 64

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2016Layer Normalizationcited by 5,669
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2023A Survey of Large Language Modelscited by 2,576
  10. 2024Qwen2 Technical Reportcited by 2,120
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  12. 2020GLU Variants Improve Transformercited by 1,804
38 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  2. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  8. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  9. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  10. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  11. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  12. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  13. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  14. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  15. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  16. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  17. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  18. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  19. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  20. 2024DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Datacited by 234
  21. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  22. 2024Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systemscited by 164
  23. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  24. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  25. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  26. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 130
  27. 2024Lean Workbook: A large-scale Lean problem set formalized from natural language math problemscited by 122
  28. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  29. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  30. 2024Stable LM 2 1.6B Technical Reportcited by 90
  31. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 89
  32. 2024Lean-STaR: Learning to Interleave Thinking and Provingcited by 68
  33. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67
  34. 2024Enhancing LLM Reasoning with Reward-guided Tree Searchcited by 58
  35. 2023A Theory on Adam Instability in Large-Scale Machine Learningcited by 56
  36. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 52
  37. 2023WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Modelscited by 50
  38. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 45

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Mid-Training of Large Language Models: A Surveycited by 21
  2. 2025PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computingcited by 4
  3. 2025Motif 2 12.7B technical reportcited by 4
  4. 2025PCMind-2.1-Kaiyuan-2B Technical Reportcited by 2
  5. 2026daVinci-LLM:Towards the Science of Pretrainingcited by 0
  6. 2026Towards Efficient LLMs Annealing with Principled Sample Selectioncited by 0
  7. 2026From Copying to Composition: How Language Models Develop In-Context Learningcited by 0