# Curvature-Sensitive Predictive Coding with Approximate Laplace Monte Carlo

arXiv: https://arxiv.org/abs/2303.04976 · HTML: https://mlsanity.com/paper/2303.04976/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 58 references that are in the map.
- First posted on arXiv 9 Mar 2023.

## Builds on (most cited first, 18 of 18)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning](https://mlsanity.com/paper/2104.04975/) (2021)
- [Predictive Coding Approximates Backprop along Arbitrary Computation Graphs](https://mlsanity.com/paper/2006.04182/) (2020)
- [Predictive Coding: a Theoretical and Experimental Review](https://mlsanity.com/paper/2107.12979/) (2021)
- [RNADE: The real-valued neural autoregressive density-estimator](https://mlsanity.com/paper/1306.0186/) (2013)
- [The free energy principle for action and perception: A mathematical review](https://mlsanity.com/paper/1705.09156/) (2017)
- [Invariance Learning in Deep Neural Networks with Differentiable Laplace Approximations](https://mlsanity.com/paper/2202.10638/) (2022)
- [Hybrid Predictive Coding: Inferring, Fast and Slow](https://mlsanity.com/paper/2204.02169/) (2022)
- [Variational Laplace Autoencoders](https://mlsanity.com/paper/2211.17267/) (2022)

## Built on by (most cited first, 3 of 3)

- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [Predictive Coding as a Neuromorphic Alternative to Backpropagation: A Critical Evaluation](https://mlsanity.com/paper/2304.02658/) (2023)
- [Predictive Coding Networks and Inference Learning: Tutorial and Survey](https://mlsanity.com/paper/2407.04117/) (2024)

## Abstract (arXiv)

Predictive coding (PC) accounts of perception now form one of the dominant computational theories of the brain, where they prescribe a general algorithm for inference and learning over hierarchical latent probabilistic models. Despite this, they have enjoyed little export to the broader field of machine learning, where comparative generative modelling techniques have flourished. In part, this has been due to the poor performance of models trained with PC when evaluated by both sample quality and marginal likelihood. By adopting the perspective of PC as a variational Bayes algorithm under the Laplace approximation, we identify the source of these deficits to lie in the exclusion of an associated Hessian term in the PC objective function, which would otherwise regularise the sharpness of the probability landscape and prevent over-certainty in the approximate posterior. To remedy this, we make three primary contributions: we begin by suggesting a simple Monte Carlo estimated evidence lower bound which relies on sampling from the Hessian-parameterised variational posterior. We then derive a novel block diagonal approximation to the full Hessian matrix that has lower memory requirements and favourable mathematical properties. Lastly, we present an algorithm that combines our method with standard PC to reduce memory complexity further. We evaluate models trained with our approach against the standard PC framework on image benchmark datasets. Our approach produces higher log-likelihoods and qualitatively better samples that more closely capture the diversity of the data-generating distribution.