# Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training

arXiv: https://arxiv.org/abs/2509.06518 · HTML: https://mlsanity.com/paper/2509.06518/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 8 Sep 2025.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [A Survey on Model Compression for Large Language Models](https://mlsanity.com/paper/2308.07633/) (2023)
- [OpenELM: An Efficient Language Model Family with Open Training and Inference Framework](https://mlsanity.com/paper/2404.14619/) (2024)
- [Model Compression and Efficient Inference for Large Language Models: A Survey](https://mlsanity.com/paper/2402.09748/) (2024)
- [Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training](https://mlsanity.com/paper/2405.15319/) (2024)
- [The Impact of Depth on Compositional Generalization in Transformer Language Models](https://mlsanity.com/paper/2310.19956/) (2023)
- [Is Bigger and Deeper Always Better? Probing LLaMA Across Scales and Layers](https://mlsanity.com/paper/2312.04333/) (2023)
- [Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective](https://mlsanity.com/paper/2502.14770/) (2025)
- [LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions](https://mlsanity.com/paper/2505.23811/) (2025)
- [DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMs](https://mlsanity.com/paper/2407.11030/) (2024)
- [Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training](https://mlsanity.com/paper/2502.03460/) (2025)

## Built on by (most cited first, 2 of 2)

- [Tapered Language Models](https://mlsanity.com/paper/2606.23670/) (2026)
- [Variable-Width Transformers](https://mlsanity.com/paper/2606.18246/) (2026)

## Abstract (arXiv)

Transformer-based language models traditionally use uniform (isotropic) layer sizes, yet they ignore the diverse functional roles that different depths can play and their computational capacity needs. Building on Layer-Wise Scaling (LWS) and pruning literature, we introduce three new LWS variants - Framed, Reverse, and Crown - that redistribute FFN widths and attention heads via two or three-point linear interpolation in the pre-training stage. We present the first systematic ablation of LWS and its variants, on a fixed budget of 180M parameters, trained on 5B tokens. All models converge to similar losses and achieve better performance compared to an equal-cost isotropic baseline, without a substantial decrease in training throughput. This work represents an initial step into the design space of layer-wise architectures for pre-training, but future work should scale experiments to orders of magnitude more tokens and parameters to fully assess their potential.