# Constrained Parameter Inference as a Principle for Learning

arXiv: https://arxiv.org/abs/2203.13203 · HTML: https://mlsanity.com/paper/2203.13203/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 50 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [Explainable Deep Learning: A Field Guide for the Uninitiated](https://mlsanity.com/paper/2004.14545/) (2020)
- [Dendritic cortical microcircuits approximate the backpropagation algorithm](https://mlsanity.com/paper/1810.11393/) (2018)
- [Reducing Overfitting in Deep Networks by Decorrelating Representations](https://mlsanity.com/paper/1511.06068/) (2015)
- [Deep Learning without Weight Transport](https://mlsanity.com/paper/1904.05391/) (2019)
- [Decorrelated Batch Normalization](https://mlsanity.com/paper/1804.08450/) (2018)
- [How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation](https://mlsanity.com/paper/1407.7906/) (2014)
- [A Hebbian/Anti-Hebbian Neural Network for Linear Subspace Learning: A Derivation from Multidimensional Scaling of Streaming Data](https://mlsanity.com/paper/1503.00669/) (2015)
- [Equilibrium Propagation with Continual Weight Updates](https://mlsanity.com/paper/2005.04168/) (2020)
- [GAIT-prop: A biologically plausible learning rule derived from backpropagation of error](https://mlsanity.com/paper/2006.06438/) (2020)
- [Spike-based causal inference for weight alignment](https://mlsanity.com/paper/1910.01689/) (2019)
- [ZORB: A Derivative-Free Backpropagation Algorithm for Neural Networks](https://mlsanity.com/paper/2011.08895/) (2020)
- [Overcoming the Weight Transport Problem via Spike-Timing-Dependent Weight Inference](https://mlsanity.com/paper/2003.03988/) (2020)
- [Do retinal ganglion cells project natural scenes to their principal subspace and whiten them?](https://mlsanity.com/paper/1612.03483/) (2016)
- [Gradient-adjusted Incremental Target Propagation Provides Effective Credit Assignment in Deep Neural Networks](https://mlsanity.com/paper/2102.11598/) (2021)

## Built on by (most cited first, 9 of 9)

- [Efficient Deep Learning with Decorrelated Backpropagation](https://mlsanity.com/paper/2405.02385/) (2024)
- [Correlations Are Ruining Your Gradient Descent](https://mlsanity.com/paper/2407.10780/) (2024)
- [Gradient-Free Training of Recurrent Neural Networks using Random Perturbations](https://mlsanity.com/paper/2405.08967/) (2024)
- [Ornstein-Uhlenbeck Adaptation as a Mechanism for Learning in Brains and Machines](https://mlsanity.com/paper/2410.13563/) (2024)
- [Node Perturbation Can Effectively Train Multi-Layer Neural Networks](https://mlsanity.com/paper/2310.00965/) (2023)
- [Noise-based Local Learning using Stochastic Magnetic Tunnel Junctions](https://mlsanity.com/paper/2412.12783/) (2024)
- [Efficient Deep Reinforcement Learning with Predictive Processing Proximal Policy Optimization](https://mlsanity.com/paper/2211.06236/) (2022)
- [Real-Time Decorrelation-Based Anomaly Detection for Multivariate Time Series](https://mlsanity.com/paper/2507.07559/) (2025)
- [Decorrelation Speeds Up Vision Transformers](https://mlsanity.com/paper/2510.14657/) (2025)

## Abstract (arXiv)

Learning in neural networks is often framed as a problem in which targeted error signals are directly propagated to parameters and used to produce updates that induce more optimal network behaviour. Backpropagation of error (BP) is an example of such an approach and has proven to be a highly successful application of stochastic gradient descent to deep neural networks. We propose constrained parameter inference (COPI) as a new principle for learning. The COPI approach assumes that learning can be set up in a manner where parameters infer their own values based upon observations of their local neuron activities. We find that this estimation of network parameters is possible under the constraints of decorrelated neural inputs and top-down perturbations of neural states for credit assignment. We show that the decorrelation required for COPI allows learning at extremely high learning rates, competitive with that of adaptive optimizers, as used by BP. We further demonstrate that COPI affords a new approach to feature analysis and network compression. Finally, we argue that COPI may shed new light on learning in biological networks given the evidence for decorrelation in the brain.