mlsanityData to 14 Oct 2026

Learning to Optimize by Differentiable Programming

arXiv 2601.16510 ↗First posted 23 Jan 2026cs.MS cs.LG math.OCcited by 1

Not on the map. Why?

Abstract (arXiv)

Solving massive-scale optimization problems requires scalable first-order methods with low per-iteration cost. This tutorial highlights a shift in optimization: using differentiable programming not only to execute algorithms but to learn how to design them. Modern frameworks such as PyTorch, TensorFlow, and JAX enable this paradigm through efficient automatic differentiation. Embedding first-order methods within these systems allows end-to-end training that improves convergence and solution quality. Guided by Fenchel-Rockafellar duality, the tutorial demonstrates how duality-informed iterative schemes such as the alternating direction method of multipliers, and the primal-dual hybrid gradient can be learned and adapted through representative case studies.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 86 of its 213 references that are in the map. First posted on arXiv 23 Jan 2026.

Builds on

86 of 213 references.

References · 86

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2021Convex optimizationcited by 2,515
  4. 2016An overview of gradient descent optimization algorithmscited by 1,033
  5. 2015Automatic differentiation in machine learning: a surveycited by 855
  6. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
  7. 2016CVXPY: A Python-Embedded Modeling Language for Convex Optimizationcited by 641
  8. 2018Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizoncited by 623
  9. 2016Learning to learn by gradient descent by gradient descentcited by 620
  10. 2019Deep Equilibrium Modelscited by 615
  11. 2017OptNet: Differentiable Optimization as a Layer in Neural Networkscited by 599
  12. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 567
38 more
  1. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  2. 2019Differentiable Convex Optimization Layerscited by 465
  3. 2018Efficient Neural Network Robustness Certification with General Activation Functionscited by 445
  4. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  5. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 258
  6. 2014cuDNN: Efficient Primitives for Deep Learningcited by 245
  7. 2016Hyperparameter optimization with approximate gradientcited by 233
  8. 2021Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Complete and Incomplete Neural Network Robustness Verificationcited by 228
  9. 2019Differentiation of Blackbox Combinatorial Solverscited by 226
  10. 2021Learning to Optimize: A Primer and A Benchmarkcited by 208
  11. 2019Towards Stable and Efficient Training of Verifiably Robust Neural Networkscited by 205
  12. 2020Fast Differentiable Sorting and Rankingcited by 197
  13. 2018Differentiable MPC for End-to-end Planning and Controlcited by 189
  14. 2020Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifierscited by 187
  15. 2017A Rewriting System for Convex Optimization Problemscited by 179
  16. 2018Semidefinite relaxations for certifying robustness to adversarial examplescited by 179
  17. 2017OSQP: An Operator Splitting Solver for Quadratic Programscited by 161
  18. 2021End-to-End Constrained Optimization Learning: A Surveycited by 157
  19. 2023Decision-Focused Learning: Foundations, State of the Art, Benchmark and Future Opportunitiescited by 130
  20. 2015Gradient Estimation Using Stochastic Computation Graphscited by 121
  21. 2016On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimizationcited by 112
  22. 2017TensorFlow Distributionscited by 106
  23. 2015Global Convergence of ADMM in Nonconvex Nonsmooth Optimizationcited by 98
  24. 2019A Differentiable Programming System to Bridge Machine Learning and Scientific Computingcited by 83
  25. 2019Deep Declarative Networks: A New Hopecited by 78
  26. 2017CNN-Based Projected Gradient Descent for Consistent Image Reconstructioncited by 77
  27. 2020Lagrangian Duality for Constrained Deep Learningcited by 74
  28. 2019Differentiating Through a Cone Programcited by 73
  29. 2022Theseus: A Library for Differentiable Nonlinear Optimizationcited by 65
  30. 2021Introduction to Neural Network Verificationcited by 63
  31. 2022Self-Supervised Primal-Dual Learning for Constrained Optimizationcited by 61
  32. 2020Differentially Private and Fair Deep Learning: A Lagrangian Dual Approachcited by 53
  33. 2018Training verified learners with learned verifierscited by 50
  34. 2019Scalable Semidefinite Programmingcited by 36
  35. 2021Practical Large-Scale Linear Programming using Primal-Dual Hybrid Gradientcited by 32
  36. 2018Linear Convergence of the Primal-Dual Gradient Method for Convex-Concave Saddle Point Problems without Strong Convexitycited by 29
  37. 2014Playing with Duality: An Overview of Recent Primal-Dual Approaches for Solving Large-Scale Optimization Problemscited by 25
  38. 2018On the Differentiability of the Solution to Convex Optimization Problemscited by 23

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Difference-of-Convex Regularization for Graph Learning by Differentiable Programmingcited by 0