mlsanityData to 14 Oct 2026

How to guess a gradient

arXiv 2312.04709 ↗First posted 7 Dec 2023cs.LG cs.NEcited by 7

Not on the map. Why?

Abstract (arXiv)

How much can you say about the gradient of a neural network without computing a loss or knowing the label? This may sound like a strange question: surely the answer is "very little." However, in this paper, we show that gradients are more structured than previously thought. Gradients lie in a predictable low-dimensional subspace which depends on the network architecture and incoming features. Exploiting this structure can significantly improve gradient-free optimization schemes based on directional derivatives, which have struggled to scale beyond small networks trained on toy datasets. We study how to narrow the gap in optimization performance between methods that calculate exact gradients and those that use directional derivatives. Furthermore, we highlight new challenges in overcoming the large gap between optimizing with exact gradients and guessing the gradients.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 21 of its 42 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

21 of 42 references.

References · 21

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2022Visual Prompt Tuningcited by 1,681
  6. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  7. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  8. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  9. 2016Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagationcited by 259
  10. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  11. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  12. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 137
9 more
  1. 2023Stable and low-precision training for large-scale vision-language modelscited by 75
  2. 2022Gradients without Backpropagationcited by 73
  3. 2022Scaling Forward Gradient With Local Lossescited by 65
  4. 2019Low-Memory Neural Network Training: A Technical Reportcited by 57
  5. 2023Can Forward Gradient Match Backpropagation?cited by 26
  6. 2020Randomized Automatic Differentiationcited by 23
  7. 2022Optimization without Backpropagationcited by 11
  8. 2023One Forward is Enough for Neural Network Training via Likelihood Ratio Methodcited by 11
  9. 2020Sideways: Depth-Parallel Training of Video Modelscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  2. 2025Position: Curvature Matrices Should Be Democratized via Linear Operatorscited by 9
  3. 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 5
  4. 2026Credit Assignment via Neural Manifold Noise Correlationcited by 1
  5. 2025Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functionscited by 0
  6. 2025Towards Scalable Backpropagation-Free Gradient Estimationcited by 0
  7. 2026Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Samplingcited by 0