mlsanityData to 7 Oct 2026

OptNet: Differentiable Optimization as a Layer in Neural Networks

arXiv 1703.00443 ↗First posted 1 Mar 2017cs.LG cs.AI math.OCcited by 588

Not on the map. Why?

Abstract (arXiv)

This paper presents OptNet, a network architecture that integrates optimization problems (here, specifically in the form of quadratic programs) as individual layers in larger end-to-end trainable deep networks. These layers encode constraints and complex dependencies between the hidden states that traditional convolutional and fully-connected layers often cannot capture. We explore the foundations for such an architecture: we show how techniques from sensitivity analysis, bilevel optimization, and implicit differentiation can be used to exactly differentiate through these layers and with respect to layer parameters; we develop a highly efficient solver for these layers that exploits fast GPU-based batch solves within a primal-dual interior point method, and which provides backpropagation gradients with virtually no additional cost on top of the solve; and we highlight the application of these approaches in several problems. In one notable example, the method is learns to play mini-Sudoku (4x4) given just input and output games, with no a-priori information about the rules of the game; this highlights the ability of OptNet to learn hard constraints better than other neural architectures.

Key facts. Cited by 588 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Mar 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 90; 2022: 92; 2023: 89; 2024: 84; 2025: 93; 2026: 140.

588 papers.

Sort:

By area:

  1. 2022Transformers learn in-context by gradient descentcited by 714
  2. 2021Implicit Behavioral Cloningcited by 442
  3. 2021Efficient and Modular Implicit Differentiationcited by 199
  4. 2021DC3: A learning method for optimization with hard constraintscited by 181
  5. 2021End-to-End Constrained Optimization Learning: A Surveycited by 154
  6. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 151
  7. 2023Decision-Focused Learning: Foundations, State of the Art, Benchmark and Future Opportunitiescited by 129
  8. 2021Model-Based Domain Generalizationcited by 103
  9. 2021Implicit MLE: Backpropagating Through Discrete Exponential Family Distributionscited by 97
  10. 2023Uncovering mesa-optimization algorithms in Transformerscited by 94
  11. 2022Tutorial on amortized optimizationcited by 83
  12. 2021A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learningcited by 82
38 more
  1. 2023A Survey of Contextual Optimization Methods for Decision Making under Uncertaintycited by 81
  2. 2021Stiff Neural Ordinary Differential Equationscited by 77
  3. 2021Large-Scale Wasserstein Gradient Flowscited by 76
  4. 2021Is Attention Better Than Matrix Decomposition?cited by 75
  5. 2021Differentiable Learning Under Triagecited by 74
  6. 2021On Training Implicit Modelscited by 73
  7. 2021Stabilizing Equilibrium Models by Jacobian Regularizationcited by 71
  8. 2023SafeDiffuser: Safe Planning with Diffusion Probabilistic Modelscited by 69
  9. 2022Neural Conservation Laws: A Divergence-Free Perspectivecited by 67
  10. 2022Theseus: A Library for Differentiable Nonlinear Optimizationcited by 64
  11. 2022Deep Equilibrium Optical Flow Estimationcited by 63
  12. 2021Decision-Focused Learning: Through the Lens of Learning to Rankcited by 61
  13. 2022Improving Object-centric Learning with Query Optimizationcited by 60
  14. 2021Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modelingcited by 59
  15. 2021Strategic Classification Made Practicalcited by 58
  16. 2021An End-to-End Differentiable Framework for Contact-Aware Robot Designcited by 53
  17. 2021Fast and Feature-Complete Differentiable Physics for Articulated Rigid Bodies with Contactcited by 50
  18. 2022Joint Distribution Matters: Deep Brownian Distance Covariance for Few-Shot Classificationcited by 50
  19. 2021CombOptNet: Fit the Right NP-Hard Problem by Learning Integer Programming Constraintscited by 49
  20. 2021Learnable Graph Matching: Incorporating Graph Partitioning with Deep Feature Learning for Multiple Object Trackingcited by 47
  21. 2022Object Representations as Fixed Points: Training Iterative Refinement Algorithms with Implicit Differentiationcited by 46
  22. 2021Safe Reinforcement Learning Using Robust Control Barrier Functionscited by 44
  23. 2021Nonsmooth Implicit Differentiation for Machine Learning and Optimizationcited by 43
  24. 2022Deep Equilibrium Approaches to Diffusion Modelscited by 42
  25. 2021Adversarial Robustness with Semi-Infinite Constrained Learningcited by 39
  26. 2025MesaNet: Sequence Modeling by Locally Optimal Test-Time Trainingcited by 37
  27. 2024Learning Iterative Reasoning through Energy Diffusioncited by 36
  28. 2022Learning differentiable solvers for systems with hard constraintscited by 34
  29. 2022PyEPO: A PyTorch-based End-to-End Predict-then-Optimize Library for Linear and Integer Programmingcited by 32
  30. 2022Learning Iterative Reasoning through Energy Minimizationcited by 32
  31. 2024PatternBoost: Constructions in Mathematics with a Little Help from AIcited by 32
  32. 2021Enforcing Policy Feasibility Constraints through Differentiable Projection for Energy Optimizationcited by 31
  33. 2023RAYEN: Imposition of Hard Convex Constraints on Neural Networkscited by 30
  34. 2024Physics-Informed Neural Networks with Hard Linear Equality Constraintscited by 30
  35. 2022Backpropagation through Combinatorial Algorithms: Identity with Projection Workscited by 29
  36. 2024HardNet: Hard-Constrained Neural Networks with Universal Approximation Guaranteescited by 29
  37. 2025Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimizationcited by 29
  38. 2021Implicit Normalizing Flowscited by 28

50 most cited of 588 All 588 (JSON)