mlsanityData to 14 Oct 2026

MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

arXiv 2404.06395 ↗First posted 9 Apr 2024cs.CL cs.LGcited by 723

Not on the map. Why?

Cited by 112 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The burgeoning interest in developing Large Language Models (LLMs) with up to trillion parameters has been met with concerns regarding resource efficiency and practical expense, particularly given the immense cost of experimentation. This scenario underscores the importance of exploring the potential of Small Language Models (SLMs) as a resource-efficient alternative. In this context, we introduce MiniCPM, specifically the 1.2B and 2.4B non-embedding parameter variants, not only excel in their respective categories but also demonstrate capabilities on par with 7B-13B LLMs. While focusing on SLMs, our approach exhibits scalability in both model and data dimensions for future LLM research. Regarding model scaling, we employ extensive model wind tunnel experiments for stable and optimal scaling. For data scaling, we introduce a Warmup-Stable-Decay (WSD) learning rate scheduler (LRS), conducive to continuous training and domain adaptation. We present an in-depth analysis of the intriguing training dynamics that occurred in the WSD LRS. With WSD LRS, we are now able to efficiently study data-model scaling law without extensive retraining experiments on both axes of model and data, from which we derive the much higher compute optimal data-model ratio than Chinchilla Optimal. Additionally, we introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-MoE and MiniCPM-128K, whose excellent performance further cementing MiniCPM's foundation in diverse SLM applications. MiniCPM models are available publicly at https://github.com/OpenBMB/MiniCPM .

Key facts. Cited by 723 papers in the map (data through 14 Oct 2026). Builds on 62 of its 73 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

62 of 73 references.

References · 62

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
38 more
  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2023Mistral 7Bcited by 3,164
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  13. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  14. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  15. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  16. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  17. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  18. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  19. 2023StarCoder: may the source be with you!cited by 799
  20. 2024Yi: Open Foundation Models by 01.AIcited by 719
  21. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  22. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  23. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  24. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  25. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  26. 2023Textbooks Are All You Needcited by 569
  27. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  28. 2023The Falcon Series of Open Language Modelscited by 535
  29. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  30. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  31. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  32. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  33. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  34. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
  35. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  36. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  37. 2020Query-Key Normalization for Transformerscited by 280
  38. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 190; 2025: 303; 2026: 230.

723 papers.

Sort:

By area:

  1. 2024Qwen2.5 Technical Reportcited by 4,942
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  3. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  4. 2025Large Language Diffusion Modelscited by 912
  5. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  6. 2025Kimi K2: Open Agentic Intelligencecited by 567
  7. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  8. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  9. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  10. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  11. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  12. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
38 more
  1. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 197
  2. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 186
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  4. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 175
  5. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  6. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  7. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  8. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
  9. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  10. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  11. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  12. 2026Kimi K3: Open Frontier Intelligencecited by 138
  13. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 134
  14. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Modelscited by 133
  15. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 130
  16. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 124
  17. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 117
  18. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  19. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 113
  20. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
  21. 2024EuroLLM: Multilingual Language Models for Europecited by 106
  22. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  23. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
  24. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 100
  25. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 99
  26. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 97
  27. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  28. 2025Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMscited by 96
  29. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  30. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  31. 2024Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligencecited by 86
  32. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 83
  33. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 82
  34. 2025REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjectscited by 80
  35. 2025Pretraining Large Language Models with NVFP4cited by 76
  36. 2024How Does Critical Batch Size Scale in Pre-training?cited by 73
  37. 2024Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Modelscited by 72
  38. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72

50 most cited of 723 All 723 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models
  2. 2025Where to Begin: Efficient Pretraining via Subnetwork Selection and Distillation
  3. 2025Generalizing Scaling Laws for Dense and Sparse Large Language Models
  4. 2025Small Language Models: Architectures, Techniques, Evaluation, Problems and Future Adaptation
  5. 2025LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
  6. 2024Small Language Models: Survey, Measurements, and Insights
  7. 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexity
  8. 2024A Survey of Small Language Models