# Theoretical Foundations of Scaling Law in Familial Models

arXiv: https://arxiv.org/abs/2512.23407 · HTML: https://mlsanity.com/paper/2512.23407/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 21 references that are in the map.
- First posted on arXiv 29 Dec 2025.

## Builds on (most cited first, 16 of 16)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)
- [Scaling Laws for Fine-Grained Mixture of Experts](https://mlsanity.com/paper/2402.07871/) (2024)
- [Resolving Discrepancies in Compute-Optimal Scaling of Language Models](https://mlsanity.com/paper/2406.19146/) (2024)
- [Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs](https://mlsanity.com/paper/2402.10517/) (2024)
- [AI Flow at the Network Edge](https://mlsanity.com/paper/2411.12469/) (2024)
- [Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models](https://mlsanity.com/paper/2507.17702/) (2025)
- [AI Flow: Perspectives, Scenarios, and Approaches](https://mlsanity.com/paper/2506.12479/) (2025)
- [Reconciling Kaplan and Chinchilla Scaling Laws](https://mlsanity.com/paper/2406.12907/) (2024)
- [Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inference](https://mlsanity.com/paper/2503.12926/) (2025)
- [ThriftLLM: On Cost-Effective Selection of Large Language Models for Classification Queries](https://mlsanity.com/paper/2501.04901/) (2025)
- [LLM Applications: Current Paradigms and the Next Frontier](https://mlsanity.com/paper/2503.04596/) (2025)

## Built on by (most cited first, 3 of 3)

- [A Real-Time Privacy-Preserving Behavior Recognition System via Edge-Cloud Collaboration](https://mlsanity.com/paper/2601.22938/) (2026)
- [Ruyi2.5 Technical Report](https://mlsanity.com/paper/2603.17311/) (2026)
- [Silicon Bureaucracy and AI Test-Oriented Education: Contamination Sensitivity and Score Confidence in LLM Benchmarks](https://mlsanity.com/paper/2603.21636/) (2026)

## Abstract (arXiv)

Neural scaling laws have become foundational for optimizing large language model (LLM) training, yet they typically assume a single dense model output. This limitation effectively overlooks "Familial models, a transformative paradigm essential for realizing ubiquitous intelligence across heterogeneous device-edge-cloud hierarchies. Transcending static architectures, familial models integrate early exits with relay-style inference to spawn G deployable sub-models from a single shared backbone. In this work, we theoretically and empirically extend the scaling law to capture this "one-run, many-models" paradigm by introducing Granularity (G) as a fundamental scaling variable alongside model size (N) and training tokens (D). To rigorously quantify this relationship, we propose a unified functional form L(N, D, G) and parameterize it using large-scale empirical runs. Specifically, we employ a rigorous IsoFLOP experimental design to strictly isolate architectural impact from computational scale. Across fixed budgets, we systematically sweep model sizes (N) and granularities (G) while dynamically adjusting tokens (D). This approach effectively decouples the marginal cost of granularity from the benefits of scale, ensuring high-fidelity parameterization of our unified scaling law. Our results reveal that the granularity penalty follows a multiplicative power law with an extremely small exponent. Theoretically, this bridges fixed-compute training with dynamic architectures. Practically, it validates the "train once, deploy many" paradigm, demonstrating that deployment flexibility is achievable without compromising the compute-optimality of dense baselines.