# PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training

arXiv: https://arxiv.org/abs/2606.06470 · HTML: https://mlsanity.com/paper/2606.06470/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 102 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Builds on (most cited first, 25 of 25)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Muon is Scalable for LLM Training](https://mlsanity.com/paper/2502.16982/) (2025)
- [Spectral Norm Regularization for Improving the Generalizability of Deep Learning](https://mlsanity.com/paper/1705.10941/) (2017)
- [Language models scale reliably with over-training and on downstream tasks](https://mlsanity.com/paper/2403.08540/) (2024)
- [The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithm](https://mlsanity.com/paper/2505.16932/) (2025)
- [A Spectral Condition for Feature Learning](https://mlsanity.com/paper/2310.17813/) (2023)
- [Fantastic Pretraining Optimizers and Where to Find Them](https://mlsanity.com/paper/2509.02046/) (2025)
- [mHC: Manifold-Constrained Hyper-Connections](https://mlsanity.com/paper/2512.24880/) (2025)
- [Muon Optimizes Under Spectral Norm Constraints](https://mlsanity.com/paper/2506.15054/) (2025)
- [A Note on the Convergence of Muon](https://mlsanity.com/paper/2502.02900/) (2025)
- [Micro-Batch Training with Batch-Channel Normalization and Weight Standardization](https://mlsanity.com/paper/1903.10520/) (2019)
- [Muon Outperforms Adam in Tail-End Associative Memory Learning](https://mlsanity.com/paper/2509.26030/) (2025)
- [Controlled LLM Training on Spectral Sphere](https://mlsanity.com/paper/2601.08393/) (2026)
- [Training Transformers with Enforced Lipschitz Constants](https://mlsanity.com/paper/2507.13338/) (2025)
- [Rethinking Language Model Scaling under Transferable Hypersphere Optimization](https://mlsanity.com/paper/2603.28743/) (2026)
- [Enhancing LLM Training via Spectral Clipping](https://mlsanity.com/paper/2603.14315/) (2026)
- [Learning in Compact Spaces with Approximately Normalized Transformer](https://mlsanity.com/paper/2505.22014/) (2025)
- [Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation](https://mlsanity.com/paper/2605.12492/) (2026)
- [On the Convergence Analysis of Muon](https://mlsanity.com/paper/2505.23737/) (2025)
- [POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformation](https://mlsanity.com/paper/2603.05500/) (2026)

## Built on by (most cited first, 1 of 1)

- [ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training](https://mlsanity.com/paper/2610.06116/) (2026)

## Abstract (arXiv)

We propose a preconditioning (PC) layer, a weight parameterization via polynomial preconditioner that ensures stable weight conditioning throughout LLM training. The PC module reshapes the singular-value spectrum of weight matrices via low-degree polynomial preconditioning. After training, the preconditioned weights can be merged back into the original architecture, incurring no inference overhead. We demonstrate the advantage of the proposed PC layer over standard transformers in Llama-1B pre-training, for both the AdamW and Muon optimizers. Theoretically, we justify this spectrum-control principle by proving that uniformly bounding each layer's singular values ensures geometric convergence of gradient descent to global minima, for certain deep linear networks. Our code is available at https://github.com/Empath-aln/PC-layer.