# Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models

arXiv: https://arxiv.org/abs/2506.10972 · HTML: https://mlsanity.com/paper/2506.10972/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 45 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://mlsanity.com/paper/2408.00724/) (2024)
- [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [A Survey on Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2407.06204/) (2024)
- [Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://mlsanity.com/paper/2401.00448/) (2023)
- [The Quantization Model of Neural Scaling](https://mlsanity.com/paper/2303.13506/) (2023)
- [Observational Scaling Laws and the Predictability of Language Model Performance](https://mlsanity.com/paper/2405.10938/) (2024)
- [Scaling Laws for Precision](https://mlsanity.com/paper/2411.04330/) (2024)
- [Chinchilla Scaling: A replication attempt](https://mlsanity.com/paper/2404.10102/) (2024)
- [How Feature Learning Can Improve Neural Scaling Laws](https://mlsanity.com/paper/2409.17858/) (2024)
- [Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models](https://mlsanity.com/paper/2501.12370/) (2025)
- [Scaling Laws for Downstream Task Performance of Large Language Models](https://mlsanity.com/paper/2402.04177/) (2024)
- [Scaling Laws for Predicting Downstream Performance in LLMs](https://mlsanity.com/paper/2410.08527/) (2024)
- [A Simple Model of Inference Scaling Laws](https://mlsanity.com/paper/2410.16377/) (2024)
- [Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data](https://mlsanity.com/paper/2411.06646/) (2024)
- [Scaling Laws For Dense Retrieval](https://mlsanity.com/paper/2403.18684/) (2024)
- [Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCo](https://mlsanity.com/paper/2503.09799/) (2025)
- [Scaling Law for Recommendation Models: Towards General-purpose User Representations](https://mlsanity.com/paper/2111.11294/) (2021)

## Built on by (most cited first, 6 of 6)

- [Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients](https://mlsanity.com/paper/2606.25008/) (2026)
- [Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection](https://mlsanity.com/paper/2604.22753/) (2026)
- [Skaling: Chinchilla's Exponents Meet Kaplan's Coupling](https://mlsanity.com/paper/2608.07222/) (2026)
- [Configuration-to-Performance Scaling Law with Neural Ansatz](https://mlsanity.com/paper/2602.10300/) (2026)
- [Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits](https://mlsanity.com/paper/2603.22339/) (2026)
- [Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation](https://mlsanity.com/paper/2605.08541/) (2026)

## Abstract (arXiv)

Training Large Language Models (LLMs) is prohibitively expensive, creating a critical scaling gap where insights from small-scale experiments often fail to transfer to resource-intensive production systems, thereby hindering efficient innovation. To bridge this, we introduce Farseer, a novel and refined scaling law offering enhanced predictive accuracy across scales. By systematically constructing a model loss surface L(N,D), Farseer achieves a significantly better fit to empirical data than prior laws (e.g., Chinchilla's law). Our methodology yields accurate, robust, and highly generalizable predictions, demonstrating excellent extrapolation capabilities, improving upon Chinchilla's law by reducing extrapolation error by 433%. This allows for the reliable evaluation of competing training strategies across all (N,D) settings, enabling conclusions from small-scale ablation studies to be confidently extrapolated to predict large-scale performance. Furthermore, Farseer provides new insights into optimal compute allocation, better reflecting the nuanced demands of modern LLM training. To validate our approach, we trained an extensive suite of approximately 1,000 LLMs across diverse scales and configurations, consuming roughly 3 million NVIDIA H100 GPU hours. We are comprehensively open-sourcing all models, data, results, and logs at https://github.com/Farseer-Scaling-Law/Farseer to foster further research.