# Bilevel Programs Meet Deep Learning: A Unifying View on Inference Learning Methods

arXiv: https://arxiv.org/abs/2105.07231 · HTML: https://mlsanity.com/paper/2105.07231/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 15 May 2021.

## Builds on (most cited first, 21 of 21)

- [Meta-Learning with Implicit Gradients](https://mlsanity.com/paper/1909.04630/) (2019)
- [Bilevel Programming for Hyperparameter Optimization and Meta-Learning](https://mlsanity.com/paper/1806.04910/) (2018)
- [Gradient-based Hyperparameter Optimization through Reversible Learning](https://mlsanity.com/paper/1502.03492/) (2015)
- [Optimizing Millions of Hyperparameters by Implicit Differentiation](https://mlsanity.com/paper/1911.02590/) (2019)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [Predictive Coding Approximates Backprop along Arbitrary Computation Graphs](https://mlsanity.com/paper/2006.04182/) (2020)
- [How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation](https://mlsanity.com/paper/1407.7906/) (2014)
- [Deep Declarative Networks: A New Hope](https://mlsanity.com/paper/1909.04866/) (2019)
- [Towards Biologically Plausible Deep Learning](https://mlsanity.com/paper/1502.04156/) (2015)
- [A Theoretical Framework for Target Propagation](https://mlsanity.com/paper/2006.14331/) (2020)
- [Biologically Motivated Algorithms for Propagating Local Target Representations](https://mlsanity.com/paper/1805.11703/) (2018)
- [Convergent Block Coordinate Descent for Training Tikhonov Regularized Deep Neural Networks](https://mlsanity.com/paper/1711.07354/) (2017)
- [Lifted Neural Networks](https://mlsanity.com/paper/1805.01532/) (2018)
- [Fenchel Lifted Networks: A Lagrange Relaxation of Neural Network Training](https://mlsanity.com/paper/1811.08039/) (2018)
- [Lifted Proximal Operator Machines](https://mlsanity.com/paper/1811.01501/) (2018)
- [Techniques for Gradient Based Bilevel Optimization with Nonsmooth Lower Level Problems](https://mlsanity.com/paper/1602.07080/) (2016)
- [Contrastive Learning for Lifted Networks](https://mlsanity.com/paper/1905.02507/) (2019)
- [Proximal Backpropagation](https://mlsanity.com/paper/1706.04638/) (2017)
- [Lifted Regression/Reconstruction Networks](https://mlsanity.com/paper/2005.03452/) (2020)
- [Truncated Inference for Latent Variable Optimization Problems: Application to Robust Estimation and Learning](https://mlsanity.com/paper/2003.05886/) (2020)

## Built on by (most cited first, 5 of 5)

- [Dual Propagation: Accelerating Contrastive Hebbian Learning with Dyadic Neurons](https://mlsanity.com/paper/2302.01228/) (2023)
- [Towards training digitally-tied analog blocks via hybrid gradient computation](https://mlsanity.com/paper/2409.03306/) (2024)
- [Training a Predictive Coding Network on ImageNet using Equilibrium Propagation](https://mlsanity.com/paper/2606.03584/) (2026)
- [Two Tales of Single-Phase Contrastive Hebbian Learning](https://mlsanity.com/paper/2402.08573/) (2024)
- [Strengthening the Internal Adversarial Robustness in Lifted Neural Networks](https://mlsanity.com/paper/2503.07818/) (2025)

## Abstract (arXiv)

In this work we unify a number of inference learning methods, that are proposed in the literature as alternative training algorithms to the ones based on regular error back-propagation. These inference learning methods were developed with very diverse motivations, mainly aiming to enhance the biological plausibility of deep neural networks and to improve the intrinsic parallelism of training methods. We show that these superficially very different methods can all be obtained by successively applying a particular reformulation of bilevel optimization programs. As a by-product it becomes also evident that all considered inference learning methods include back-propagation as a special case, and therefore at least approximate error back-propagation in typical settings. Finally, we propose Fenchel back-propagation, that replaces the propagation of infinitesimal corrections performed in standard back-propagation with finite targets as the learning signal. Fenchel back-propagation can therefore be seen as an instance of learning via explicit target propagation.