mlsanityData to 14 Oct 2026

CAST: History-Adaptive Spectral Clipping for LLM Pretraining

arXiv 2610.19761 ↗First posted 14 Oct 2026cs.LGcited by 0

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Soft spectral clipping (SSC) reduces gradient singular values, but its effect depends on the gradient scale and clipping threshold. We introduce CAST (Clipping with Adaptive Scales and Thresholds), a history-adaptive extension of SSC. AdamW's previous second moment sets row and column scales. An exponential moving average of the clipped output's estimated spectral norm sets each matrix's threshold. CAST restores the original coordinates before the AdamW update. We train a mixture-of-experts (MoE) model with 19.08B total parameters for 83.89B tokens. CAST achieves lower training and mean validation losses than nine gradient-processing baselines, with the fewest relative gradient spikes. It reaches the best baseline's final smoothed training loss in 7.2% fewer steps.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 14 of its 27 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

14 of 27 references.

References · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2017Mixed Precision Trainingcited by 1,156
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  7. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  8. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  9. 2024Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Expertscited by 32
  10. 2025ZClip: Adaptive Spike Mitigation for LLM Pre-Trainingcited by 10
  11. 2026Enhancing LLM Training via Spectral Clippingcited by 10
  12. 2025GradPower: Powering Gradients for Faster Language Model Pre-Trainingcited by 8
2 more
  1. 2025AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clippingcited by 6
  2. 2025GradientStabilizer:Fix the Norm, Not the Gradientcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy
  2. 2026ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training
  3. 2026The Lifecycle of the Spectral Edge: From Gradient Learning to Weight-Decay Compression
  4. 2026Intrinsic Gradient Suppression for Label-Noise Prompt Tuning in Vision-Language Models
  5. 2024Learning Continually by Spectral Regularization
  6. 2025Bayesian Natural Gradient Fine-Tuning of CLIP Models via Kalman Filtering
  7. 2026Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights
  8. 2026S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations