# A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains

arXiv: https://arxiv.org/abs/2310.13849 · HTML: https://mlsanity.com/paper/2310.13849/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 90 references that are in the map.
- First posted on arXiv 20 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification](https://mlsanity.com/paper/2010.05300/) (2020)
- [Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway](https://mlsanity.com/paper/1411.6422/) (2014)
- [Polar Transformer Networks](https://mlsanity.com/paper/1709.01889/) (2017)
- [Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision](https://mlsanity.com/paper/1608.03425/) (2016)
- [FOVEA: Foveated Image Magnification for Autonomous Navigation](https://mlsanity.com/paper/2108.12102/) (2021)
- [Biologically Inspired Mechanisms for Adversarial Robustness](https://mlsanity.com/paper/2006.16427/) (2020)
- [FoveaTer: Foveated Transformer for Image Classification](https://mlsanity.com/paper/2105.14173/) (2021)
- [Peripheral Vision Transformer](https://mlsanity.com/paper/2206.06801/) (2022)
- [Attention for Fine-Grained Categorization](https://mlsanity.com/paper/1412.7054/) (2014)
- [Object Detection Through Exploration With A Foveated Visual Field](https://mlsanity.com/paper/1408.0814/) (2014)
- [On the use of Cortical Magnification and Saccades as Biological Proxies for Data Augmentation](https://mlsanity.com/paper/2112.07173/) (2021)
- [Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks](https://mlsanity.com/paper/2202.00838/) (2022)
- [Joint rotational invariance and adversarial training of a dual-stream Transformer yields state of the art Brain-Score for Area V4](https://mlsanity.com/paper/2203.06649/) (2022)
- [Learning Scene Gist with Convolutional Neural Networks to Improve Object Recognition](https://mlsanity.com/paper/1803.01967/) (2018)
- [Human Eyes Inspired Recurrent Neural Networks are More Robust Against Adversarial Noises](https://mlsanity.com/paper/2206.07282/) (2022)
- [Predictive Coding for Dynamic Visual Processing: Development of Functional Hierarchy in a Multiple Spatio-Temporal Scales RNN Model](https://mlsanity.com/paper/1708.00812/) (2017)

## Built on by (most cited first, 12 of 12)

- [Cognitive Disentanglement for Referring Multi-Object Tracking](https://mlsanity.com/paper/2503.11496/) (2025)
- [Achieving More Human Brain-Like Vision via Human EEG Representational Alignment](https://mlsanity.com/paper/2401.17231/) (2024)
- [Bridging AI and Science: Implications from a Large-Scale Literature Analysis of AI4Science](https://mlsanity.com/paper/2412.09628/) (2024)
- [Teaching CORnet Human fMRI Representations for Enhanced Model-Brain Alignment](https://mlsanity.com/paper/2407.10414/) (2024)
- [L-WISE: Boosting Human Visual Category Learning Through Model-Based Image Selection and Enhancement](https://mlsanity.com/paper/2412.09765/) (2024)
- [DecoFuse: Decomposing and Fusing the "What", "Where", and "How" for Brain-Inspired fMRI-to-Video Decoding](https://mlsanity.com/paper/2504.00432/) (2025)
- [NEvo: Neural-Guided Evolutionary Video Synthesis for Dynamic Visual Selectivity](https://mlsanity.com/paper/2607.02317/) (2026)
- [BIMM: Brain Inspired Masked Modeling for Video Representation Learning](https://mlsanity.com/paper/2405.12757/) (2024)
- [Revealing the Learning Process in Reinforcement Learning Agents Through Attention-Oriented Metrics](https://mlsanity.com/paper/2406.14324/) (2024)
- [Quantifying task-relevant representational similarity using decision variable correlation](https://mlsanity.com/paper/2506.02164/) (2025)
- [A biologically inspired separable learning vision model for real-time traffic object perception in Dark](https://mlsanity.com/paper/2509.05012/) (2025)
- [What-Where Transformer: A Slot-Centric Visual Backbone for Concurrent Representation and Localization](https://mlsanity.com/paper/2605.12021/) (2026)

## Abstract (arXiv)

The human visual system uses two parallel pathways for spatial processing and object recognition. In contrast, computer vision systems tend to use a single feedforward pathway, rendering them less robust, adaptive, or efficient than human vision. To bridge this gap, we developed a dual-stream vision model inspired by the human eyes and brain. At the input level, the model samples two complementary visual patterns to mimic how the human eyes use magnocellular and parvocellular retinal ganglion cells to separate retinal inputs to the brain. At the backend, the model processes the separate input patterns through two branches of convolutional neural networks (CNN) to mimic how the human brain uses the dorsal and ventral cortical pathways for parallel visual processing. The first branch (WhereCNN) samples a global view to learn spatial attention and control eye movements. The second branch (WhatCNN) samples a local view to represent the object around the fixation. Over time, the two branches interact recurrently to build a scene representation from moving fixations. We compared this model with the human brains processing the same movie and evaluated their functional alignment by linear transformation. The WhereCNN and WhatCNN branches were found to differentially match the dorsal and ventral pathways of the visual cortex, respectively, primarily due to their different learning objectives. These model-based results lead us to speculate that the distinct responses and representations of the ventral and dorsal streams are more influenced by their distinct goals in visual attention and object recognition than by their specific bias or selectivity in retinal inputs. This dual-stream model takes a further step in brain-inspired computer vision, enabling parallel neural networks to actively explore and understand the visual surroundings.