mlsanityData to 7 Oct 2026

Learning to learn by gradient descent by gradient descent

arXiv 1606.04474 ↗First posted 14 Jun 2016cs.NE cs.LGcited by 616

Not on the map. Why?

Abstract (arXiv)

The move from hand-designed features to learned features in machine learning has been wildly successful. In spite of this, optimization algorithms are still designed by hand. In this paper we show how the design of an optimization algorithm can be cast as a learning problem, allowing the algorithm to learn to exploit structure in the problems of interest in an automatic way. Our learned algorithms, implemented by LSTMs, outperform generic, hand-designed competitors on the tasks for which they are trained, and also generalize well to new tasks with similar structure. We demonstrate this on a number of tasks, including simple convex problems, training neural networks, and styling images with neural art.

Key facts. Cited by 616 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 0; 2021: 133; 2022: 138; 2023: 103; 2024: 82; 2025: 80; 2026: 78.

616 papers.

Sort:

By area:

  1. 2022Transformers learn in-context by gradient descentcited by 714
  2. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  3. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  4. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  5. 2024Titans: Learning to Memorize at Test Timecited by 303
  6. 2021Gradient Matching for Domain Generalizationcited by 258
  7. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 248
  8. 2021Learning to Optimize: A Primer and A Benchmarkcited by 206
  9. 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 162
  10. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 151
  11. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 138
  12. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
38 more
  1. 2021DeFRCN: Decoupled Faster R-CNN for Few-Shot Object Detectioncited by 106
  2. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  3. 2021MetaSAug: Meta Semantic Augmentation for Long-Tailed Visual Recognitioncited by 99
  4. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  5. 2022Learning from Few Examples: A Summary of Approaches to Few-Shot Learningcited by 86
  6. 2022Learning to Learn with Generative Models of Neural Network Checkpointscited by 85
  7. 2022Tutorial on amortized optimizationcited by 83
  8. 2021Gradients are Not All You Needcited by 82
  9. 2021MetaAvatar: Learning Animatable Clothed Human Models from Few Depth Imagescited by 78
  10. 2023Machine learning with data assimilation and uncertainty quantification for dynamical systems: a reviewcited by 74
  11. 2021Partial Is Better Than All: Revisiting Fine-tuning Strategy for Few-shot Learningcited by 72
  12. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  13. 2023Permutation Equivariant Neural Functionalscited by 69
  14. 2022Neural Conservation Laws: A Divergence-Free Perspectivecited by 67
  15. 2023MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Taskscited by 66
  16. 2023Advances and Challenges in Meta-Learning: A Technical Reviewcited by 65
  17. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  18. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  19. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 57
  20. 2022VeLO: Training Versatile Learned Optimizers by Scaling Upcited by 57
  21. 2023How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligencecited by 55
  22. 2021MetaAlign: Coordinating Domain Alignment and Classification for Unsupervised Domain Adaptationcited by 54
  23. 2022SparsePose: Sparse-View Camera Pose Regression and Refinementcited by 53
  24. 2025End-to-End Test-Time Training for Long Contextcited by 53
  25. 2021Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategiescited by 52
  26. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variablescited by 50
  27. 2023Neural Functional Transformerscited by 50
  28. 2021Incremental Few-Shot Instance Segmentationcited by 49
  29. 2021Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learningcited by 49
  30. 2022Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiationcited by 46
  31. 2021Bootstrapped Meta-Learningcited by 45
  32. 2026EvoX: Meta-Evolution for Automated Discoverycited by 45
  33. 2021Categorical Foundations of Gradient-Based Learningcited by 44
  34. 2022Discovering Evolution Strategies via Meta-Black-Box Optimizationcited by 44
  35. 2023Improving Language Plasticity via Pretraining with Active Forgettingcited by 43
  36. 2021Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspectivecited by 39
  37. 2022Transformers are Meta-Reinforcement Learnerscited by 39
  38. 2023An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learningcited by 39

50 most cited of 616 All 616 (JSON)