# Can Language Models Discover Scaling Laws?

arXiv: https://arxiv.org/abs/2507.21184 · HTML: https://mlsanity.com/paper/2507.21184/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 77 references that are in the map.
- First posted on arXiv 27 Jul 2025.

## Builds on (most cited first, 30 of 53)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [AlphaEvolve: A coding agent for scientific and algorithmic discovery](https://mlsanity.com/paper/2506.13131/) (2025)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [DeepSeek LLM: Scaling Open-Source Language Models with Longtermism](https://mlsanity.com/paper/2401.02954/) (2024)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [Scaling Data-Constrained Language Models](https://mlsanity.com/paper/2305.16264/) (2023)

## Built on by (most cited first, 13 of 13)

- [CodeEvolve: an open source evolutionary coding agent for algorithmic discovery and optimization](https://mlsanity.com/paper/2510.14150/) (2025)
- [NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents](https://mlsanity.com/paper/2510.07172/) (2025)
- [Evolution Fine-Tuning: Learning to Discover Across 371 Optimization Tasks](https://mlsanity.com/paper/2606.29082/) (2026)
- [Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection](https://mlsanity.com/paper/2604.22753/) (2026)
- [Evolutionary Multi-Task Optimization for LLM-Guided Program Discovery](https://mlsanity.com/paper/2605.22613/) (2026)
- [Configuration-to-Performance Scaling Law with Neural Ansatz](https://mlsanity.com/paper/2602.10300/) (2026)
- [Improved Upper Bounds for Slicing the Hypercube](https://mlsanity.com/paper/2602.16807/) (2026)
- [Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design](https://mlsanity.com/paper/2608.03501/) (2026)
- [Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search](https://mlsanity.com/paper/2609.01431/) (2026)
- [Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits](https://mlsanity.com/paper/2603.22339/) (2026)
- [Practical Scaling Laws: Converting Compute into Performance in a Data-Constrained World](https://mlsanity.com/paper/2605.09189/) (2026)
- [Algorithmic algorithm development with LLMs: A Case Study on LLM-Usage for Contraction Order Optimization in Tensor Networks](https://mlsanity.com/paper/2606.01975/) (2026)
- [SLDBench v2: Active Scaling Law Discovery](https://mlsanity.com/paper/2610.16297/) (2026)

## Abstract (arXiv)

Discovering scaling laws for predicting model performance at scale is a fundamental and open-ended challenge, mostly reliant on slow, case specific human experimentation. To investigate the potential for LLMs to automate this process, we collect over 5,000 experiments from existing literature and curate eight diverse scaling law discovery tasks. While existing agents struggle to produce accurate law formulas, this paper introduces SLDAgent, an evolution-based agent that co-optimize the scaling law model and the parameters, enabling it to autonomously explore complex relationships between variables. For the first time, we demonstrates that SLDAgent can automatically discover laws that exhibit consistently more accurate extrapolation than their established, human-derived counterparts across all tasks. Through comprehensive analysis, we elucidate why these discovered laws are superior and verify their practical utility in both pretraining and finetuning applications. This work establishes a new paradigm for agentic scientific discovery, showing that AI systems can understand their own scaling behavior, and can contribute novel and practical knowledge back to the research community.