# Gradient Shaping Beyond Clipping: A Functional Perspective on Update Magnitude Control

arXiv: https://arxiv.org/abs/2510.01578 · HTML: https://mlsanity.com/paper/2510.01578/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 2 Oct 2025.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Understanding Gradient Clipping in Private SGD: A Geometric Perspective](https://mlsanity.com/paper/2006.15429/) (2020)
- [The Heavy-Tail Phenomenon in SGD](https://mlsanity.com/paper/2006.04740/) (2020)
- [Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learning](https://mlsanity.com/paper/2202.03599/) (2022)
- [Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clipping](https://mlsanity.com/paper/2005.10785/) (2020)
- [Revisiting Gradient Clipping: Stochastic bias and tight convergence guarantees](https://mlsanity.com/paper/2305.01588/) (2023)
- [Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecasting](https://mlsanity.com/paper/2507.02939/) (2025)
- [From Gradient Clipping to Normalization for Heavy Tailed SGD](https://mlsanity.com/paper/2410.13849/) (2024)
- [Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothness](https://mlsanity.com/paper/2102.06489/) (2021)
- [SepPrune: Structured Pruning for Efficient Deep Speech Separation](https://mlsanity.com/paper/2505.12079/) (2025)
- [SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Models](https://mlsanity.com/paper/2410.14720/) (2024)
- [Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training](https://mlsanity.com/paper/2410.23922/) (2024)
- [COMAE: COMprehensive Attribute Exploration for Zero-shot Hashing](https://mlsanity.com/paper/2402.16424/) (2024)
- [Gradient-based Sampling: An Adaptive Importance Sampling for Least-squares](https://mlsanity.com/paper/1803.00841/) (2018)
- [Modular MeanFlow: Towards Stable and Scalable One-Step Generative Modeling](https://mlsanity.com/paper/2508.17426/) (2025)
- [On the Convergence of DP-SGD with Adaptive Clipping](https://mlsanity.com/paper/2412.19916/) (2024)
- [GradientStabilizer:Fix the Norm, Not the Gradient](https://mlsanity.com/paper/2502.17055/) (2025)
- [MOVER: Multimodal Optimal Transport with Volume-based Embedding Regularization](https://mlsanity.com/paper/2508.12149/) (2025)
- [Clipped SGD Algorithms for Performative Prediction: Tight Bounds for Clipping Bias and Remedies](https://mlsanity.com/paper/2404.10995/) (2024)
- [Layer-wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learning](https://mlsanity.com/paper/2503.14205/) (2025)

## Built on by (most cited first, 1 of 1)

- [AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Training](https://mlsanity.com/paper/2601.11864/) (2026)

## Abstract (arXiv)

Gradient clipping is widely used to stabilize deep network training, but its formulation as a hard, fixed threshold limits flexibility and ignores gradient distribution dynamics. We propose SPAMP (Statistical Per-layer Adaptive Modulation and Projection), a unified framework that generalizes clipping into smooth, per-layer gradient shaping. SPAMP tracks local gradient statistics, dynamically estimates thresholds, and applies power-based transformations to modulate update magnitudes in a differentiable manner. This perspective recasts clipping and warmup as dual mechanisms for controlling the effective update scale ηt |gt|, offering a principled alternative to rigid heuristics. Extensive experiments across image and language tasks demonstrate that SPAMP improves stability, convergence, and robustness over existing methods.