mlsanityData to 7 Oct 2026

Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster

arXiv 2304.03208 ↗First posted 6 Apr 2023cs.LG cs.CLcited by 106

Not on the map. Why?

Abstract (arXiv)

We study recent research advances that improve large language models through efficient pre-training and scaling, and open datasets and tools. We combine these advances to introduce Cerebras-GPT, a family of open compute-optimal language models scaled from 111M to 13B parameters. We train Cerebras-GPT models on the Eleuther Pile dataset following DeepMind Chinchilla scaling rules for efficient pre-training (highest accuracy for a given compute budget). We characterize the predictable power-law scaling and compare Cerebras-GPT with other publicly-available models to show all Cerebras-GPT models have state-of-the-art training efficiency on both pre-training and downstream objectives. We describe our learnings including how Maximal Update Parameterization (μP) can further improve large model scaling, improving accuracy and hyperparameter predictability at scale. We release our pre-trained models and code, making this paper the first open and reproducible work comparing compute-optimal model scaling to models trained on fixed dataset sizes. Cerebras-GPT models are available on HuggingFace: https://huggingface.co/cerebras.

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Builds on 42 of its 57 references that are in the map. First posted on arXiv 6 Apr 2023.

Builds on

42 of 57 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
30 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2020GLU Variants Improve Transformercited by 1,724
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2017Mixed Precision Trainingcited by 1,135
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2018Model Cards for Model Reportingcited by 1,068
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  12. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  13. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  14. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  15. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  16. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  17. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  18. 2022UL2: Unifying Language Learning Paradigmscited by 292
  19. 2018An Empirical Model of Large-Batch Trainingcited by 268
  20. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 260
  21. 2020Understanding the Difficulty of Training Transformerscited by 210
  22. 2016Deep Information Propagationcited by 194
  23. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  24. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  25. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  26. 2017Mean Field Residual Networks: On the Edge of Chaoscited by 83
  27. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  28. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 45
  29. 2021The Stability-Efficiency Dilemma: Investigating Sequence Length Warmup for Training GPT Modelscited by 36
  30. 2021Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learningcited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 29; 2024: 36; 2025: 24; 2026: 17.

106 papers.

Sort:

By area:

  1. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  2. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  3. 2023The Falcon Series of Open Language Modelscited by 526
  4. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  5. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  6. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  7. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  8. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  9. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  10. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  11. 2023Generative Language Models Exhibit Social Identity Biasescited by 107
  12. 2023A Spectral Condition for Feature Learningcited by 101
38 more
  1. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  2. 2024Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arenacited by 76
  3. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  4. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 74
  5. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  6. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  7. 2024How Does Critical Batch Size Scale in Pre-training?cited by 68
  8. 2025Distillation Scaling Lawscited by 59
  9. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 53
  10. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  11. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
  12. 2025Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-trainingcited by 41
  13. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 36
  14. 2024KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Modelscited by 36
  15. 2024Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Modelscited by 35
  16. 2024MobiLlama: Towards Accurate and Lightweight Fully Transparent GPTcited by 34
  17. 2025VaViM and VaVAM: Autonomous Driving through Video Generative Modelingcited by 34
  18. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 34
  19. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33
  20. 2023HuRef: HUman-REadable Fingerprint for Large Language Modelscited by 32
  21. 2024Contextual Feature Extraction Hierarchies Converge in Large Language Models and the Braincited by 32
  22. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  23. 2024Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Schedulercited by 26
  24. 2023From Text to Source: Results in Detecting Large Language Model-Generated Contentcited by 25
  25. 2025Get Experience from Practice: LLM Agents with Record & Replaycited by 22
  26. 2023Training and inference of large language models using 8-bit floating pointcited by 20
  27. 2024Causal Estimation of Memorisation Profilescited by 20
  28. 2025Weight Decay may matter more than muP for Learning Rate Transfer in Practicecited by 20
  29. 2023BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Modelcited by 19
  30. 2025Gemstones: A Model Suite for Multi-Faceted Scaling Lawscited by 19
  31. 2024Foundational Large Language Models for Materials Researchcited by 18
  32. 2023TarGEN: Targeted Data Generation with Large Language Modelscited by 17
  33. 2024LimGen: Probing the LLMs for Generating Suggestive Limitations of Research Paperscited by 16
  34. 2025MuLoCo: Muon is a practical inner optimizer for DiLoCocited by 16
  35. 2023TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decompositioncited by 15
  36. 2024ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Lawscited by 15
  37. 2025The Energy Cost of Reasoning: Analyzing Energy Usage in LLMs with Test-time Computecited by 14
  38. 2024TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguesecited by 12

50 most cited of 106 All 106 (JSON)