# How does the primate brain combine generative and discriminative computations in vision?

arXiv: https://arxiv.org/abs/2401.06005 · HTML: https://mlsanity.com/paper/2401.06005/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 138 references that are in the map.
- First posted on arXiv 11 Jan 2024.

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation](https://mlsanity.com/paper/2007.04954/) (2020)
- [Generative Modelling With Inverse Heat Dissipation](https://mlsanity.com/paper/2206.13397/) (2022)
- [DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning](https://mlsanity.com/paper/2006.08381/) (2020)
- [Physion: Evaluating Physical Prediction from Vision in Humans and Machines](https://mlsanity.com/paper/2106.08261/) (2021)
- [Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway](https://mlsanity.com/paper/1411.6422/) (2014)
- [Iterative Amortized Inference](https://mlsanity.com/paper/1807.09356/) (2018)
- [Recurrence is required to capture the representational dynamics of the human visual system](https://mlsanity.com/paper/1903.05946/) (2019)
- [On the role of planning in model-based deep reinforcement learning](https://mlsanity.com/paper/2011.04021/) (2020)
- [Efficient Iterative Amortized Inference for Learning Symmetric and Disentangled Multi-Object Representations](https://mlsanity.com/paper/2106.03630/) (2021)
- [Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputs](https://mlsanity.com/paper/2306.11536/) (2023)
- [The implications of perception as probabilistic inference for correlated neural variability during behavior](https://mlsanity.com/paper/1409.0257/) (2014)
- [Beyond linear regression: mapping models in cognitive neuroscience should align with research goals](https://mlsanity.com/paper/2208.10668/) (2022)
- [Control as Hybrid Inference](https://mlsanity.com/paper/2007.05838/) (2020)
- [What is the dynamical regime of cerebral cortex?](https://mlsanity.com/paper/1908.10101/) (2019)
- [3D Shape Perception Integrates Intuitive Physics and Analysis-by-Synthesis](https://mlsanity.com/paper/2301.03711/) (2023)

## Built on by (most cited first, 4 of 4)

- [Principles of Visual Tokens for Efficient Video Understanding](https://mlsanity.com/paper/2411.13626/) (2024)
- [Bidirectional predictive coding](https://mlsanity.com/paper/2505.23415/) (2025)
- [How does the brain compute with probabilities?](https://mlsanity.com/paper/2409.02709/) (2024)
- [Is Generation Required for Data-Efficient Perception?](https://mlsanity.com/paper/2512.08854/) (2025)

## Abstract (arXiv)

Vision is widely understood as an inference problem. However, two contrasting conceptions of the inference process have each been influential in research on biological vision as well as the engineering of machine vision. The first emphasizes bottom-up signal flow, describing vision as a largely feedforward, discriminative inference process that filters and transforms the visual information to remove irrelevant variation and represent behaviorally relevant information in a format suitable for downstream functions of cognition and behavioral control. In this conception, vision is driven by the sensory data, and perception is direct because the processing proceeds from the data to the latent variables of interest. The notion of "inference" in this conception is that of the engineering literature on neural networks, where feedforward convolutional neural networks processing images are said to perform inference. The alternative conception is that of vision as an inference process in Helmholtz's sense, where the sensory evidence is evaluated in the context of a generative model of the causal processes giving rise to it. In this conception, vision inverts a generative model through an interrogation of the evidence in a process often thought to involve top-down predictions of sensory data to evaluate the likelihood of alternative hypotheses. The authors include scientists rooted in roughly equal numbers in each of the conceptions and motivated to overcome what might be a false dichotomy between them and engage the other perspective in the realm of theory and experiment. The primate brain employs an unknown algorithm that may combine the advantages of both conceptions. We explain and clarify the terminology, review the key empirical evidence, and propose an empirical research program that transcends the dichotomy and sets the stage for revealing the mysterious hybrid algorithm of primate vision.