# ZClip: Adaptive Spike Mitigation for LLM Pre-Training

arXiv: https://arxiv.org/abs/2504.02507 · HTML: https://mlsanity.com/paper/2504.02507/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 13 references that are in the map.
- First posted on arXiv 3 Apr 2025.

## Builds on (most cited first, 6 of 6)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [RegMix: Data Mixture as Regression for Language Model Pre-training](https://mlsanity.com/paper/2407.01492/) (2024)
- [Spike No More: Stabilizing the Pre-training of Large Language Models](https://mlsanity.com/paper/2312.16903/) (2023)
- [AutoClip: Adaptive Gradient Clipping for Source Separation Networks](https://mlsanity.com/paper/2007.14469/) (2020)

## Built on by (most cited first, 10 of 10)

- [QVGen: Pushing the Limit of Quantized Video Generative Models](https://mlsanity.com/paper/2505.11497/) (2025)
- [Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention](https://mlsanity.com/paper/2606.20945/) (2026)
- [MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration](https://mlsanity.com/paper/2602.01734/) (2026)
- [oboro: Text-to-Image Synthesis on Limited Data using Flow-based Diffusion Transformer with MMH Attention](https://mlsanity.com/paper/2511.08168/) (2025)
- [Active Inference with a Self-Prior in the Mirror-Mark Task](https://mlsanity.com/paper/2604.09673/) (2026)
- [A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Models](https://mlsanity.com/paper/2604.16809/) (2026)
- [From Syntax to Semantics: Unveiling the Emergence of Chirality in SMILES Translation Models](https://mlsanity.com/paper/2605.09949/) (2026)
- [OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers](https://mlsanity.com/paper/2607.04033/) (2026)
- [Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction](https://mlsanity.com/paper/2609.13285/) (2026)
- [CAST: History-Adaptive Spectral Clipping for LLM Pretraining](https://mlsanity.com/paper/2610.19761/) (2026)

## Abstract (arXiv)

Training large language models (LLMs) presents numerous challenges, including gradient instability and loss spikes. These phenomena can lead to catastrophic divergence, requiring costly checkpoint restoration and data batch skipping. Traditional gradient clipping techniques, such as constant or norm-based methods, fail to address these issues effectively due to their reliance on fixed thresholds or heuristics, leading to inefficient learning and requiring frequent manual intervention. In this work, we propose ZClip, an adaptive gradient clipping algorithm that dynamically adjusts the clipping threshold based on statistical properties of gradient norms over time. Unlike prior reactive strategies, ZClip proactively adapts to training dynamics without making any prior assumptions on the scale and the temporal evolution of gradient norms. At its core, it leverages z-score-based anomaly detection to identify and mitigate large gradient spikes, preventing malignant loss spikes while not interfering with convergence otherwise. Our code is available at: https://github.com/bluorion-com/ZClip.