mlsanityData to 7 Oct 2026

PanGu-π Pro:Rethinking Optimization and Architecture for Tiny Language Models

arXiv 2402.02791 ↗First posted 5 Feb 2024cs.CL cs.AI cs.LGcited by 14

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

The power of large language models (LLMs) has been demonstrated through numerous data and computing resources. However, the application of language models on mobile devices is facing huge challenge on the computation and memory costs, that is, tiny language models with high performance are urgently required. Limited by the highly complex training process, there are many details for optimizing language models that are seldom studied carefully. In this study, based on a tiny language model with 1B parameters, we carefully design a series of empirical study to analyze the effect of each component. Three perspectives are mainly discussed, ie, neural architecture, parameter initialization, and optimization strategy. Several design formulas are empirically proved especially effective for tiny language models, including tokenizer compression, architecture tweaking, parameter inheritance and multiple-round training. Then we train PanGu-π-1B Pro and PanGu-π-1.5B Pro on 1.6T multilingual corpora, following the established formulas. Experimental results demonstrate the improved optimization and architecture yield a notable average improvement of 8.87 on benchmark evaluation sets for PanGu-π-1B Pro. Besides, PanGu-π-1.5B Pro surpasses a range of SOTA models with larger model sizes, validating its superior performance. The code is available at https://github.com/YuchuanTian/RethinkTinyLM.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 27 of its 58 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

27 of 58 references.

References · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2017Decoupled Weight Decay Regularizationcited by 1,722
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  12. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
15 more
  1. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  2. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  3. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  4. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  5. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  6. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  7. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  8. 2017Large Batch Training of Convolutional Networkscited by 529
  9. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  10. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  11. 2019Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehensioncited by 80
  12. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 55
  13. 2021FewCLUE: A Chinese Few-shot Learning Evaluation Benchmarkcited by 41
  14. 2023Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learningcited by 27
  15. 2023PanGu-π: Enhancing Language Model Architectures via Nonlinearity Compensationcited by 17

Built on by

Cited by year

Citing papers per year: 2024: 8; 2025: 6.

14 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  2. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  3. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 50
  4. 2024Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradientscited by 36
  5. 2025DeepDiver: Adaptive Search Intensity Scaling via Open-Web Reinforcement Learningcited by 23
  6. 2024Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuningcited by 20
  7. 2024Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuningcited by 14
  8. 2025Eve: Efficient Multimodal Vision Language Models with Elastic Visual Expertscited by 11
  9. 2025EfficientLLM: Scalable Pruning-Aware Pretraining for Architecture-Agnostic Edge Language Modelscited by 10
  10. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  11. 2024Creation of AI-driven Smart Spaces for Enhanced Indoor Environments -- A Surveycited by 2
  12. 2025Efficiently Building a Domain-Specific Large Language Model from Scratch: A Case Study of a Classical Chinese Large Language Modelcited by 0
2 more
  1. 2025Remodeling Semantic Relationships in Vision-Language Fine-Tuningcited by 0
  2. 2025Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLMcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
  2. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insights
  3. 2025Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs
  4. 2024Search for Efficient Large Language Models
  5. 2025Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques
  6. 2024Small Language Models: Survey, Measurements, and Insights
  7. 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing
  8. 2026Surpassing Scale by Efficiency: A Compact 135M Parameter Foundational LLM Natively Adapted for the Bangla Language