mlsanityData to 14 Oct 2026

PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training

arXiv 2606.06470 ↗First posted 4 Jun 2026cs.LG cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

We propose a preconditioning (PC) layer, a weight parameterization via polynomial preconditioner that ensures stable weight conditioning throughout LLM training. The PC module reshapes the singular-value spectrum of weight matrices via low-degree polynomial preconditioning. After training, the preconditioned weights can be merged back into the original architecture, incurring no inference overhead. We demonstrate the advantage of the proposed PC layer over standard transformers in Llama-1B pre-training, for both the AdamW and Muon optimizers. Theoretically, we justify this spectrum-control principle by proving that uniformly bounding each layer's singular values ensures geometric convergence of gradient descent to global minima, for certain deep linear networks. Our code is available at https://github.com/Empath-aln/PC-layer.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 25 of its 102 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

25 of 102 references.

References · 25

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2016Layer Normalizationcited by 5,669
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  7. 2025Muon is Scalable for LLM Trainingcited by 428
  8. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 182
  9. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  10. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 110
  11. 2023A Spectral Condition for Feature Learningcited by 109
  12. 2025Fantastic Pretraining Optimizers and Where to Find Themcited by 100
13 more
  1. 2025mHC: Manifold-Constrained Hyper-Connectionscited by 88
  2. 2025Muon Optimizes Under Spectral Norm Constraintscited by 86
  3. 2025A Note on the Convergence of Muoncited by 83
  4. 2019Micro-Batch Training with Batch-Channel Normalization and Weight Standardizationcited by 46
  5. 2025Muon Outperforms Adam in Tail-End Associative Memory Learningcited by 46
  6. 2026Controlled LLM Training on Spectral Spherecited by 31
  7. 2025Training Transformers with Enforced Lipschitz Constantscited by 20
  8. 2026Rethinking Language Model Scaling under Transferable Hypersphere Optimizationcited by 12
  9. 2026Enhancing LLM Training via Spectral Clippingcited by 10
  10. 2025Learning in Compact Spaces with Approximately Normalized Transformercited by 6
  11. 2026Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformationcited by 6
  12. 2025On the Convergence Analysis of Muoncited by 5
  13. 2026POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Trainingcited by 0