# Toward a mechanistic understanding of inference in visual cortex and diffusion models

arXiv: https://arxiv.org/abs/2607.15693 · HTML: https://mlsanity.com/paper/2607.15693/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 26 of its 80 references that are in the map.
- First posted on arXiv 17 Jul 2026.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [Generalization in diffusion models arises from geometry-adaptive harmonic representations](https://mlsanity.com/paper/2310.02557/) (2023)
- [Hierarchical Reasoning Model](https://mlsanity.com/paper/2506.21734/) (2025)
- [Implicit Deep Learning](https://mlsanity.com/paper/1908.06315/) (2019)
- [An analytic theory of creativity in convolutional diffusion models](https://mlsanity.com/paper/2412.20292/) (2024)
- [JFB: Jacobian-Free Backpropagation for Implicit Networks](https://mlsanity.com/paper/2103.12803/) (2021)
- [Residual Connections Encourage Iterative Inference](https://mlsanity.com/paper/1710.04773/) (2017)
- [On Training Implicit Models](https://mlsanity.com/paper/2111.05177/) (2021)
- [Neural Empirical Bayes](https://mlsanity.com/paper/1903.02334/) (2019)
- [Towards a Mechanistic Explanation of Diffusion Model Generalization](https://mlsanity.com/paper/2411.19339/) (2024)
- [The Sparse Manifold Transform](https://mlsanity.com/paper/1806.08887/) (2018)
- [Fixed Point Diffusion Models](https://mlsanity.com/paper/2401.08741/) (2024)
- [A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models](https://mlsanity.com/paper/2408.05927/) (2024)
- [Bag of Image Patch Embedding Behind the Success of Self-Supervised Learning](https://mlsanity.com/paper/2206.08954/) (2022)
- [Effect of top-down connections in Hierarchical Sparse Coding](https://mlsanity.com/paper/2002.00892/) (2020)
- [Diffusion Model's Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold](https://mlsanity.com/paper/2602.06021/) (2026)
- [Do Diffusion Models Learn Semantically Meaningful and Efficient Representations?](https://mlsanity.com/paper/2402.03305/) (2024)

## Abstract (arXiv)

We describe a model of perceptual inference in primary visual cortex (V1) equivalent to a minimal diffusion model whose function can be readily understood from its parameters. The model is based on sparse coding with a non-factorial prior over latent variables in the form of an unconstrained, pairwise interaction matrix, extending standard sparse coding inference to a general recurrent dynamical system. We efficiently train these recurrent dynamics using a denoising score-matching objective and implicit differentiation. After training on natural images, the learned interaction matrix mirrors the structure of horizontal connections in superficial layers of V1 that link neurons of similar orientation tuning. This model exhibits exceptionally good denoising performance, restoring image features such as extended contours amid extreme visual ambiguity, nearly matching the behavior of standard, black-box diffusion architectures in generalization regime. Owing to the model's simplicity, the network's Jacobian can be decomposed directly in terms of the interaction matrix between latent variables, revealing mechanistically how the recurrent dynamics assign high probability over a continuous family of natural structural deformations. Intriguingly, within this circuit, a large fraction of latent variables learn to disconnect from visual input altogether, essentially forming a hierarchical representation that appears to enforce global consistency among image features. Together, the model and results bridge two distinct domains: for neuroscience, it generates concrete, testable hypotheses regarding functional connectivity in recurrent neural circuits during perceptual inference tasks; for machine learning, it elucidates the internal mechanisms learned by diffusion models that allow them to generate infinitely many novel images from a finite training set.