mlsanityData to 14 Oct 2026

Gradient Shaping Beyond Clipping: A Functional Perspective on Update Magnitude Control

arXiv 2510.01578 ↗First posted 2 Oct 2025cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Gradient clipping is widely used to stabilize deep network training, but its formulation as a hard, fixed threshold limits flexibility and ignores gradient distribution dynamics. We propose SPAMP (Statistical Per-layer Adaptive Modulation and Projection), a unified framework that generalizes clipping into smooth, per-layer gradient shaping. SPAMP tracks local gradient statistics, dynamically estimates thresholds, and applies power-based transformations to modulate update magnitudes in a differentiable manner. This perspective recasts clipping and warmup as dual mechanisms for controlling the effective update scale ηt |gt|, offering a principled alternative to rigid heuristics. Extensive experiments across image and language tasks demonstrate that SPAMP improves stability, convergence, and robustness over existing methods.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

23 of 32 references.

References · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Pointer Sentinel Mixture Modelscited by 3,096
  3. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  4. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  5. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 142
  6. 2020The Heavy-Tail Phenomenon in SGDcited by 124
  7. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  8. 2020Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clippingcited by 104
  9. 2023Revisiting Gradient Clipping: Stochastic bias and tight convergence guaranteescited by 83
  10. 2025Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecastingcited by 55
  11. 2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 49
  12. 2021Stability and Convergence of Stochastic Gradient Clipping: Beyond Lipschitz Continuity and Smoothnesscited by 37
11 more
  1. 2025SepPrune: Structured Pruning for Efficient Deep Speech Separationcited by 31
  2. 2024SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Modelscited by 25
  3. 2024Analyzing & Reducing the Need for Learning Rate Warmup in GPT Trainingcited by 22
  4. 2024COMAE: COMprehensive Attribute Exploration for Zero-shot Hashingcited by 16
  5. 2018Gradient-based Sampling: An Adaptive Importance Sampling for Least-squarescited by 8
  6. 2025Modular MeanFlow: Towards Stable and Scalable One-Step Generative Modelingcited by 8
  7. 2024On the Convergence of DP-SGD with Adaptive Clippingcited by 6
  8. 2025GradientStabilizer:Fix the Norm, Not the Gradientcited by 6
  9. 2025MOVER: Multimodal Optimal Transport with Volume-based Embedding Regularizationcited by 6
  10. 2024Clipped SGD Algorithms for Performative Prediction: Tight Bounds for Clipping Bias and Remediescited by 5
  11. 2025Layer-wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Trainingcited by 0