Unifying Sign and Magnitude for Optimizing Deep Vision Networks via ThermoLion
Not on the map. Why?
Abstract (arXiv)
The training of deep vision models is fundamentally a signal recovery problem amidst high-dimensional stochastic noise. Current optimization paradigms impose a static compromise on information channel capacity. For instance, magnitude-based methods, such as AdamW, operate on the assumption that gradient norms are high-fidelity curvature signals. While this allows for precision in smooth regimes, it leads to catastrophic noise amplification when applied to rugged, non-convex landscapes. Conversely, sign-based methods (e.g., Lion) perform a radical 1-bit quantization of the gradient, which aims to provide robust regularization at the cost of discarding fine-grained descent information. We propose that optimal convergence requires neither static prior, but rather a dynamic modulation of the update bitrate. We introduce ThermoLion, a vision-centric framework that utilizes local Signal-to-Noise Ratio (SNR) gating to autonomously transition parameters between a "low-bit" exploration phase and a "high-precision" exploitation phase. Furthermore, we introduce a Momentum Alignment mechanism that detects constructive interference between historical drift and instantaneous gradients to accelerate convergence during stable trajectories. Empirical benchmarks across 12 diverse vision datasets (including CIFAR, SVHN, and GTSRB) demonstrate that ThermoLion surpasses state-of-the-art optimizers, such as AdamW and Lion, in convergence speed and terminal accuracy.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 11 of its 24 references that are in the map. First posted on arXiv 1 Dec 2025.
Builds on
11 of 24 references.
References · 11
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
- 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
- 2023Symbolic Discovery of Optimization Algorithmscited by 434
- 2018Deep Learning for Classical Japanese Literaturecited by 422
- 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
- 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
- 2017Improving Generalization Performance by Switching from Adam to SGDcited by 125
- 2019Cold Case: The Lost MNIST Digitscited by 62
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Enhancing SignSGD: Small-Batch Convergence Analysis and a Hybrid Switching Strategy
- 2026Depth Laws for the Precision Floor of Trained Neural Networks: Amplification, Residual Scaling, and a Quantization-Aware Training Paradox
- 2025ANO : Faster is Better in Noisy Landscape
- 2025CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training
- 2026Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization
- 2026SoftServe: A Scalable Quasi-Newton Method for Deep Learning
- 2026SCULPT: Training Edge Vision Models for Post-Training Quantization Readiness
- 2026RAMP: Robust Adaptive Mixed-Precision Quantization for Edge CPU Vision Models