mlsanityData to 7 Oct 2026

Learning Gradient Descent: Better Generalization and Longer Horizons

arXiv 1703.03633 ↗First posted 10 Mar 2017cs.LG cs.AIcited by 50

Not on the map. Why?

Abstract (arXiv)

Training deep neural networks is a highly nontrivial task, involving carefully selecting appropriate training algorithms, scheduling step sizes and tuning other hyperparameters. Trying different combinations can be quite labor-intensive and time consuming. Recently, researchers have tried to use deep learning algorithms to exploit the landscape of the loss function of the training problem of interest, and learn how to optimize over it in an automatic way. In this paper, we propose a new learning-to-learn model and some useful and practical tricks. Our optimizer outperforms generic, hand-crafted optimization algorithms and state-of-the-art learning-to-learn optimizers by DeepMind in many tasks. We demonstrate the effectiveness of our algorithms on a number of tasks, including deep MLPs, CNNs, and simple LSTMs.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 13; 2023: 13; 2024: 5; 2025: 10; 2026: 1.

50 papers.

Sort:

By area:

  1. 2021Learning to Optimize: A Primer and A Benchmarkcited by 206
  2. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  3. 2022Tutorial on amortized optimizationcited by 83
  4. 2021Gradients are Not All You Needcited by 82
  5. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  6. 2021Bootstrapped Meta-Learningcited by 45
  7. 2022Automated Dynamic Algorithm Configurationcited by 37
  8. 2022Practical tradeoffs between memory, compute, and performance in learned optimizerscited by 32
  9. 2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
  10. 2022Gradient-based Bi-level Optimization for Deep Learning: A Surveycited by 26
  11. 2023Improving physics-informed neural networks with meta-learned optimizationcited by 19
  12. 2022Symbolic Learning to Optimize: Towards Interpretability and Scalabilitycited by 16
38 more
  1. 2023Towards Constituting Mathematical Structures for Learning to Optimizecited by 13
  2. 2021Learn2Hop: Learned Optimization on Rough Landscapescited by 12
  3. 2023Learning to Optimize for Reinforcement Learningcited by 12
  4. 2024Artificial Intelligence for Operations Research: Revolutionizing the Operations Research Processcited by 11
  5. 2022Learning to Optimize in Model Predictive Controlcited by 10
  6. 2022A Simple Guard for Learned Optimizerscited by 9
  7. 2023Mnemosyne: Learning to Train Transformers with Transformerscited by 9
  8. 2023Learning to Generalize Provably in Learning to Optimizecited by 8
  9. 2023Deep Model Predictive Optimizationcited by 7
  10. 2023Stochastic Unrolled Federated Learningcited by 6
  11. 2022Learning to Optimize with Dynamic Mode Decompositioncited by 5
  12. 2023M-L2O: Towards Generalizable Learning-to-Optimize by Test-Time Fast Self-Adaptationcited by 4
  13. 2021Model-Agnostic Meta-Attack: Towards Reliable Evaluation of Adversarial Robustnesscited by 3
  14. 2022Learning to Optimize Quasi-Newton Methodscited by 3
  15. 2024Semantic are Beacons: A Semantic Perspective for Unveiling Parameter-Efficient Fine-Tuning in Knowledge Learningcited by 3
  16. 2024Narrowing the Focus: Learned Optimizers for Pretrained Modelscited by 3
  17. 2025Efficient End-to-End Learning for Decision-Making: A Meta-Optimization Approachcited by 3
  18. 2025Learning a Zeroth-Order Optimizer for Fine-Tuning LLMscited by 3
  19. 2021Self-Learning Tuning for Post-Silicon Validationcited by 2
  20. 2021ModelPred: A Framework for Predicting Trained Model from Training Datacited by 2
  21. 2023Federated Automatic Differentiationcited by 2
  22. 2023Is Scaling Learned Optimizers Worth It? Evaluating The Value of VeLO's 4000 TPU Monthscited by 2
  23. 2023Meta-learning Optimizers for Communication-Efficient Learningcited by 2
  24. 2024From Learning to Optimize to Learning Optimization Algorithmscited by 2
  25. 2021Efficient Meta Subspace Optimizationcited by 1
  26. 2022Optimizer Amalgamationcited by 1
  27. 2025Make Optimization Once and for All with Fine-grained Guidancecited by 1
  28. 2025Accelerating Optimization via Differentiable Stopping Timecited by 1
  29. 2023Learning to optimize by multi-gradient for multi-objective optimizationcited by 0
  30. 2023Investigation into the Training Dynamics of Learned Optimizerscited by 0
  31. 2024A Learn-to-Optimize Approach for Coordinate-Wise Step Sizes for Quasi-Newton Methodscited by 0
  32. 2025Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learningcited by 0
  33. 2025Learning Provably Improves the Convergence of Gradient Descentcited by 0
  34. 2025Optimization Problem Solving Can Transition to Evolutionary Agentic Workflowscited by 0
  35. 2025Towards Robust Learning to Optimize with Theoretical Guaranteescited by 0
  36. 2025Learning from Oblivion: Predicting Knowledge Overflowed Weights via Retrodiction of Forgettingcited by 0
  37. 2025Enhancing Fractional Gradient Descent with Learned Optimizerscited by 0
  38. 2026Non-asymptotic implicit bias of logistic regression at early-stage gradient descent dynamicscited by 0