# Model-Based Deep Learning: On the Intersection of Deep Learning and Optimization

arXiv: https://arxiv.org/abs/2205.02640 · HTML: https://mlsanity.com/paper/2205.02640/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 58 references that are in the map.
- First posted on arXiv 5 May 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Compressed Sensing using Generative Models](https://mlsanity.com/paper/1703.03208/) (2017)
- [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/1911.02590/) (2019)
- [Plug-and-Play ADMM for Image Restoration: Fixed Point Convergence and Applications](https://mlsanity.com/paper/1605.01710/) (2016)
- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Model-Based Deep Learning](https://mlsanity.com/paper/2012.08405/) (2020)
- [KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics](https://mlsanity.com/paper/2107.10043/) (2021)
- [Plug and play methods for magnetic resonance imaging (long version)](https://mlsanity.com/paper/1903.08616/) (2019)
- [Deep Unfolded Robust PCA with Application to Clutter Suppression in Ultrasound](https://mlsanity.com/paper/1811.08252/) (2018)
- [Long Short-Term Memory Kalman Filters:Recurrent Neural Estimators for Pose Regularization](https://mlsanity.com/paper/1708.01885/) (2017)
- [Combining Generative and Discriminative Models for Hybrid Inference](https://mlsanity.com/paper/1906.02547/) (2019)
- [Learning Convex Optimization Control Policies](https://mlsanity.com/paper/1912.09529/) (2019)
- [Known Operator Learning and Hybrid Machine Learning in Medical Imaging -- A Review of the Past, the Present, and the Future](https://mlsanity.com/paper/2108.04543/) (2021)
- [Neural Network Detection of Data Sequences in Communication Systems](https://mlsanity.com/paper/1802.02046/) (2018)
- [Learning Convex Optimization Models](https://mlsanity.com/paper/2006.04248/) (2020)
- [Learned SPARCOM: Unfolded Deep Super-Resolution Microscopy](https://mlsanity.com/paper/2004.09270/) (2020)
- [LoRD-Net: Unfolded Deep Detection Network with Low-Resolution Receivers](https://mlsanity.com/paper/2102.02993/) (2021)
- [Deep Unrolled Recovery in Sparse Biological Imaging](https://mlsanity.com/paper/2109.14025/) (2021)
- [Neumann Networks for Inverse Problems in Imaging](https://mlsanity.com/paper/1901.03707/) (2019)
- [Fitting a Kalman Smoother to Data](https://mlsanity.com/paper/1910.08615/) (2019)
- [Learned Factor Graphs for Inference from Stationary Time Sequences](https://mlsanity.com/paper/2006.03258/) (2020)
- [Model-Based Machine Learning for Communications](https://mlsanity.com/paper/2101.04726/) (2021)
- [Learned super resolution ultrasound for improved breast lesion characterization](https://mlsanity.com/paper/2107.05270/) (2021)
- [Uncertainty in Data-Driven Kalman Filtering for Partially Known State-Space Models](https://mlsanity.com/paper/2110.04738/) (2021)
- [Ensuring DNN Solution Feasibility for Optimization Problems with Convex Constraints and Its Application to DC Optimal Power Flow Problems](https://mlsanity.com/paper/2112.08091/) (2021)

## Built on by (most cited first, 30 of 50)

- [A Survey of Contextual Optimization Methods for Decision Making under Uncertainty](https://mlsanity.com/paper/2306.10374/) (2023)
- [Scalable Optimal Transport Methods in Machine Learning: A Contemporary Survey](https://mlsanity.com/paper/2305.05080/) (2023)
- [Theoretical Perspectives on Deep Learning Methods in Inverse Problems](https://mlsanity.com/paper/2206.14373/) (2022)
- [Machine Learning for Large-Scale Optimization in 6G Wireless Networks](https://mlsanity.com/paper/2301.03377/) (2023)
- [Discriminative and Generative Learning for Linear Estimation of Random Signals [Lecture Notes]](https://mlsanity.com/paper/2206.04432/) (2022)
- [Backpropagation of Unrolled Solvers with Folded Optimization](https://mlsanity.com/paper/2301.12047/) (2023)
- [Interpretability of Machine Learning: Recent Advances and Future Prospects](https://mlsanity.com/paper/2305.00537/) (2023)
- [GSP-KalmanNet: Tracking Graph Signals via Neural-Aided Kalman Filtering](https://mlsanity.com/paper/2311.16602/) (2023)
- [SubspaceNet: Deep Learning-Aided Subspace Methods for DoA Estimation](https://mlsanity.com/paper/2306.02271/) (2023)
- [Deep Unfolding: Recent Developments, Theory, and Design Guidelines](https://mlsanity.com/paper/2512.03768/) (2025)
- [Statistical Hypothesis Testing Based on Machine Learning: Large Deviations Analysis](https://mlsanity.com/paper/2207.10939/) (2022)
- [Physics-Inspired Generative Models in Medical Imaging: A Review](https://mlsanity.com/paper/2407.10856/) (2024)
- [Optimization Guarantees of Unfolded ISTA and ADMM Networks With Smooth Soft-Thresholding](https://mlsanity.com/paper/2309.06195/) (2023)
- [Blind Channel Estimation and Joint Symbol Detection with Data-Driven Factor Graphs](https://mlsanity.com/paper/2401.12627/) (2024)
- [Implicitly Guided Design with PropEn: Match your Data to Follow the Gradient](https://mlsanity.com/paper/2405.18075/) (2024)
- [Unveiling and Mitigating Adversarial Vulnerabilities in Iterative Optimizers](https://mlsanity.com/paper/2504.19000/) (2025)
- [AI-Aided Kalman Filters](https://mlsanity.com/paper/2410.12289/) (2024)
- [Unrolled Compressed Blind-Deconvolution](https://mlsanity.com/paper/2209.14165/) (2022)
- [Neural Augmented Kalman Filtering with Bollinger Bands for Pairs Trading](https://mlsanity.com/paper/2210.15448/) (2022)
- [Uncertainty quantification for learned ISTA](https://mlsanity.com/paper/2309.07982/) (2023)
- [Limited Communications Distributed Optimization via Deep Unfolded Distributed ADMM](https://mlsanity.com/paper/2309.14353/) (2023)
- [Real-Time Model-Based Quantitative Ultrasound and Radar](https://mlsanity.com/paper/2402.10520/) (2024)
- [SINET: Sparsity-driven Interpretable Neural Network for Underwater Image Enhancement](https://mlsanity.com/paper/2409.01022/) (2024)
- [Learned Bayesian Cram'er-Rao Bound for Unknown Measurement Models Using Score Neural Networks](https://mlsanity.com/paper/2502.00724/) (2025)
- [GKNet: Graph Kalman Filtering and Model Inference via Model-based Deep Learning](https://mlsanity.com/paper/2506.22004/) (2025)
- [Plug-and-Play Latent Diffusion for Electromagnetic Inverse Scattering with Application to Brain Imaging](https://mlsanity.com/paper/2509.04860/) (2025)
- [Unrolled Graph Learning for Multi-Agent Collaboration](https://mlsanity.com/paper/2210.17101/) (2022)
- [AI-Empowered Hybrid MIMO Beamforming](https://mlsanity.com/paper/2303.01723/) (2023)
- [Deep Learning Meets Adaptive Filtering: A Stein's Unbiased Risk Estimator Approach](https://mlsanity.com/paper/2307.16708/) (2023)
- [Analyzing and Enhancing the Backward-Pass Convergence of Unrolled Optimization](https://mlsanity.com/paper/2312.17394/) (2023)

## Abstract (arXiv)

Decision making algorithms are used in a multitude of different applications. Conventional approaches for designing decision algorithms employ principled and simplified modelling, based on which one can determine decisions via tractable optimization. More recently, deep learning approaches that use highly parametric architectures tuned from data without relying on mathematical models, are becoming increasingly popular. Model-based optimization and data-centric deep learning are often considered to be distinct disciplines. Here, we characterize them as edges of a continuous spectrum varying in specificity and parameterization, and provide a tutorial-style presentation to the methodologies lying in the middle ground of this spectrum, referred to as model-based deep learning. We accompany our presentation with running examples in super-resolution and stochastic control, and show how they are expressed using the provided characterization and specialized in each of the detailed methodologies. The gains of combining model-based optimization and deep learning are demonstrated using experimental results in various applications, ranging from biomedical imaging to digital communications.