mlsanityData to 14 Oct 2026

FLM-101B: An Open LLM and How to Train It with 100K Budget

arXiv 2309.03852 ↗First posted 7 Sep 2023cs.CL cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are considered important approaches towards foundational machine intelligence, achieving remarkable success in Natural Language Processing and multimodal tasks, among others. However, the carbon footprints and financial costs originating from heavy pre-training computation is a non-negligible issue. Progressive training methods, inspired by the neurogenesis process that grows neural structures, have shown potential to accelerate LLM pre-training. However, the algorithms, implementation, and practices for progressively training LLMs beyond 100B parameters remain underexplored. In this paper, we show that our model, namely FLM-101B, trained with our growth strategy under a budget of 100K, reaches 80% of the baselines' performances with only 10% of their floating-point operations. We believe that further studies on progressive training will benefit the community by cutting down the costs and promoting green AI. The checkpoint of FLM-101B is released at https://huggingface.co/CofeAI/FLM-101B.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 67 of its 87 references that are in the map. First posted on arXiv 7 Sep 2023.

Builds on

67 of 87 references.

References · 67

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  11. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  12. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
38 more
  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  13. 2023A Survey of Large Language Modelscited by 2,576
  14. 2022Emergent Abilities of Large Language Modelscited by 2,512
  15. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  16. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  17. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  18. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  19. 2017Decoupled Weight Decay Regularizationcited by 1,831
  20. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  21. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  22. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  23. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  24. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  25. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  26. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  27. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  28. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  29. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  30. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  31. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  32. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
  33. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  34. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
  35. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  36. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  37. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  38. 2023The False Promise of Imitating Proprietary LLMscited by 263

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 15; 2025: 3.

22 papers.

Sort:

By area:

  1. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  2. 2023Efficient Large Language Models: A Surveycited by 186
  3. 2024Not All Layers of LLMs Are Necessary During Inferencecited by 96
  4. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 62
  5. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  6. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 43
  7. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 35
  8. 2024Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initializationcited by 29
  9. 2024On the Inductive Bias of Stacking Towards Improving Reasoningcited by 19
  10. 2023Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspectivecited by 17
  11. 2024Tele-FLM Technical Reportcited by 12
  12. 2023GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Lengthcited by 11
10 more
  1. 202452B to 1T: Lessons Learned via Tele-FLM Seriescited by 11
  2. 2024Rethinking Scale: The Efficacy of Fine-Tuned Open-Source LLMs in Large-Scale Reproducible Social Science Researchcited by 6
  3. 2025Scaling Diffusion Transformers Efficiently via μPcited by 6
  4. 2024Landscape-Aware Growing: The Power of a Little LAGcited by 4
  5. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  6. 2024Sketch: A Toolkit for Streamlining LLM Operationscited by 3
  7. 2024Strategic Prompting for Conversational Tasks: A Comparative Analysis of Large Language Models Across Diverse Conversational Taskscited by 3
  8. 2025Knowledge Editing for Multi-Hop Question Answering Using Semantic Analysiscited by 1
  9. 2025Beyond Sunk Costs: Boosting LLM Pre-training Efficiency via Orthogonal Growth of Mixture-of-Expertscited by 1
  10. 2024A Mean Field Ansatz for Zero-Shot Weight Transfercited by 0