mlsanityData to 14 Oct 2026

Gemstones: A Model Suite for Multi-Faceted Scaling Laws

arXiv 2502.06857 ↗First posted 7 Feb 2025cs.LG cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Scaling laws are typically fit using a family of models with a narrow range of frozen hyperparameter choices. In this work we study scaling laws using multiple architectural shapes and hyperparameter choices, highlighting their impact on resulting prescriptions. As a primary artifact of our research, we release the Gemstones: an open-source scaling law dataset, consisting of over 4000 checkpoints from transformers with up to 2 billion parameters and diverse architectural shapes; including ablations over learning rate and cooldown. Our checkpoints enable more complex studies of scaling, such as analyzing the relationship between width and depth. By examining our model suite, we find that the prescriptions of scaling laws can be highly sensitive to the experimental design process and the specific model checkpoints used during fitting.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 62 of its 80 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

62 of 80 references.

References · 62

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2023Qwen Technical Reportcited by 3,344
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
38 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  8. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  9. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  10. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  11. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  12. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  13. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  14. 2023Scaling Data-Constrained Language Modelscited by 403
  15. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  16. 20242 OLMo 2 Furiouscited by 340
  17. 2021Scaling Laws for Transfercited by 272
  18. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  19. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  20. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  21. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 173
  22. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  23. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  24. 2023Scaling Laws for Generative Mixed-Modal Language Modelscited by 134
  25. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  26. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  27. 2021Scaling Laws for Neural Machine Translationcited by 111
  28. 2022Broken Neural Scaling Lawscited by 109
  29. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  30. 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 97
  31. 2023Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Designcited by 89
  32. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 88
  33. 2024Scaling Exponents Across Parameterizations and Optimizerscited by 76
  34. 2024Chinchilla Scaling: A replication attemptcited by 72
  35. 2023Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limitcited by 66
  36. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  37. 2024Scaling Laws For Diffusion Transformerscited by 44
  38. 2023Scaling Laws for Sparsely-Connected Foundation Modelscited by 43

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 11.

20 papers.

Sort:

By area:

  1. 2025Practical Efficiency of Muon for Pretrainingcited by 80
  2. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  3. 2025Scaling Laws for Optimal Data Mixturescited by 52
  4. 2025Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrencecited by 47
  5. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 22
  6. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 21
  7. 2025Relative Scaling Laws for LLMscited by 6
  8. 2025Benchmarking Correctness and Security in Multi-Turn Code Generationcited by 4
  9. 2026Gated Recurrent Transformers: Expressive Depth through Recurrent Modulationcited by 4
  10. 2025xLSTM Scaling Laws: Competitive Performance with Linear Time-Complexitycited by 3
  11. 2026Revisiting the Shape Convention of Transformer Language Modelscited by 1
  12. 2026Variable-Width Transformerscited by 1
8 more
  1. 2026Abra: Scaling Diffusion Image Trainingcited by 1
  2. 2026DepthBench: Measuring How Residual Connections Enable More Computational Depthcited by 1
  3. 2026The Model Shape Behind Scaling Laws: A Gradient Superposition Perspectivecited by 0
  4. 2026Will Scaling Improve Social Simulation with LLMs?cited by 0
  5. 2026VAIOM: Continuous-Input, Discrete-Output Decoder-Only Financial Sequence Modelingcited by 0
  6. 2026Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Expertscited by 0
  7. 2026ScAn-Bench: Evaluating Scaling Analysis Methodologycited by 0
  8. 2026SLDBench v2: Active Scaling Law Discoverycited by 0