# Towards Two-Stream Foveation-based Active Vision Learning

arXiv: https://arxiv.org/abs/2403.15977 · HTML: https://mlsanity.com/paper/2403.15977/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 65 references that are in the map.
- First posted on arXiv 24 Mar 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Unbiased Teacher for Semi-Supervised Object Detection](https://mlsanity.com/paper/2102.09480/) (2021)
- [WIDER FACE: A Face Detection Benchmark](https://mlsanity.com/paper/1511.06523/) (2015)
- [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022)
- [Localizing Objects with Self-Supervised Transformers and no Labels](https://mlsanity.com/paper/2109.14279/) (2021)
- [Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut](https://mlsanity.com/paper/2202.11539/) (2022)
- [Attention-based Dropout Layer for Weakly Supervised Object Localization](https://mlsanity.com/paper/1908.10028/) (2019)
- [Evaluating Weakly Supervised Object Localization Methods Right](https://mlsanity.com/paper/2001.07437/) (2020)
- [Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification](https://mlsanity.com/paper/2010.05300/) (2020)
- [Instance-aware, Context-focused, and Memory-efficient Weakly Supervised Object Detection](https://mlsanity.com/paper/2004.04725/) (2020)
- [Rethinking the Route Towards Weakly Supervised Object Localization](https://mlsanity.com/paper/2002.11359/) (2020)
- [Active Object Localization with Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06015/) (2015)
- [Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modules](https://mlsanity.com/paper/2006.16981/) (2020)
- [Saccader: Improving Accuracy of Hard Attention Models for Vision](https://mlsanity.com/paper/1908.07644/) (2019)
- [Emergent Properties of Foveated Perceptual Systems](https://mlsanity.com/paper/2006.07991/) (2020)
- [DOTIE - Detecting Objects through Temporal Isolation of Events using a Spiking Architecture](https://mlsanity.com/paper/2210.00975/) (2022)
- [Semantic Relational Object Tracking](https://mlsanity.com/paper/1902.09937/) (2019)
- [EV-Planner: Energy-Efficient Robot Navigation via Event-Based Physics-Guided Neuromorphic Planner](https://mlsanity.com/paper/2307.11349/) (2023)

## Built on by (most cited first, 3 of 3)

- [DETACH: Cross-domain Learning for Long-Horizon Tasks via Mixture of Disentangled Experts](https://mlsanity.com/paper/2508.07842/) (2025)
- [On Inherent Adversarial Robustness of Active Vision Systems](https://mlsanity.com/paper/2404.00185/) (2024)
- [Task-driven Processing with Coarse-to-Fine Glimpse-based Active Perception](https://mlsanity.com/paper/2609.09025/) (2026)

## Abstract (arXiv)

Deep neural network (DNN) based machine perception frameworks process the entire input in a one-shot manner to provide answers to both "what object is being observed" and "where it is located". In contrast, the "two-stream hypothesis" from neuroscience explains the neural processing in the human visual cortex as an active vision system that utilizes two separate regions of the brain to answer the what and the where questions. In this work, we propose a machine learning framework inspired by the "two-stream hypothesis" and explore the potential benefits that it offers. Specifically, the proposed framework models the following mechanisms: 1) ventral (what) stream focusing on the input regions perceived by the fovea part of an eye (foveation), 2) dorsal (where) stream providing visual guidance, and 3) iterative processing of the two streams to calibrate visual focus and process the sequence of focused image patches. The training of the proposed framework is accomplished by label-based DNN training for the ventral stream model and reinforcement learning for the dorsal stream model. We show that the two-stream foveation-based learning is applicable to the challenging task of weakly-supervised object localization (WSOL), where the training data is limited to the object class or its attributes. The framework is capable of both predicting the properties of an object and successfully localizing it by predicting its bounding box. We also show that, due to the independent nature of the two streams, the dorsal model can be applied on its own to unseen images to localize objects from different datasets.