# FOVEA: Foveated Image Magnification for Autonomous Navigation

arXiv: https://arxiv.org/abs/2108.12102 · HTML: https://mlsanity.com/paper/2108.12102/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 47 references that are in the map.
- First posted on arXiv 27 Aug 2021.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [SSD: Single Shot MultiBox Detector](https://mlsanity.com/paper/1512.02325/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [EfficientDet: Scalable and Efficient Object Detection](https://mlsanity.com/paper/1911.09070/) (2019)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition](https://mlsanity.com/paper/1406.4729/) (2014)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks](https://mlsanity.com/paper/1312.6229/) (2013)
- [DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution](https://mlsanity.com/paper/2006.02334/) (2020)
- [Flow-Guided Feature Aggregation for Video Object Detection](https://mlsanity.com/paper/1703.10025/) (2017)
- [You Only Look One-level Feature](https://mlsanity.com/paper/2103.09460/) (2021)
- [Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference](https://mlsanity.com/paper/1912.03203/) (2019)

## Built on by (most cited first, 30 of 32)

- [Real-time Object Detection for Streaming Perception](https://mlsanity.com/paper/2203.12338/) (2022)
- [ZoomTrack: Target-aware Non-uniform Resizing for Efficient Visual Tracking](https://mlsanity.com/paper/2310.10071/) (2023)
- [SALISA: Saliency-based Input Sampling for Efficient Video Object Detection](https://mlsanity.com/paper/2204.02397/) (2022)
- [A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains](https://mlsanity.com/paper/2310.13849/) (2023)
- [JPerceiver: Joint Perception Network for Depth, Pose and Layout Estimation in Driving Scenes](https://mlsanity.com/paper/2207.07895/) (2022)
- [Learning to Zoom and Unzoom](https://mlsanity.com/paper/2303.15390/) (2023)
- [Efficient High-Resolution Deep Learning: A Survey](https://mlsanity.com/paper/2207.13050/) (2022)
- [An Empirical Analysis of Range for 3D Object Detection](https://mlsanity.com/paper/2308.04054/) (2023)
- [A Visual Active Search Framework for Geospatial Exploration](https://mlsanity.com/paper/2211.15788/) (2022)
- [A Partially Supervised Reinforcement Learning Framework for Visual Active Search](https://mlsanity.com/paper/2310.09689/) (2023)
- [Foveation in the Era of Deep Learning](https://mlsanity.com/paper/2312.01450/) (2023)
- [StreamYOLO: Real-time Object Detection for Streaming Perception](https://mlsanity.com/paper/2207.10433/) (2022)
- [Instance-Warp: Saliency Guided Image Warping for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2403.12712/) (2024)
- [CVP: Central-Peripheral Vision-Inspired Multimodal Model for Spatial Reasoning](https://mlsanity.com/paper/2512.08135/) (2025)
- [Human Eyes Inspired Recurrent Neural Networks are More Robust Against Adversarial Noises](https://mlsanity.com/paper/2206.07282/) (2022)
- [Learned Two-Plane Perspective Prior based Image Resampling for Efficient Object Detection](https://mlsanity.com/paper/2303.14311/) (2023)
- [Spatial Frequency Modulation for Semantic Segmentation](https://mlsanity.com/paper/2507.11893/) (2025)
- [SSBNet: Improving Visual Recognition Efficiency by Adaptive Sampling](https://mlsanity.com/paper/2207.11511/) (2022)
- [Looking Locally: Object-Centric Vision Transformers as Foundation Models for Efficient Segmentation](https://mlsanity.com/paper/2502.02763/) (2025)
- [Local Scale Equivariance with Latent Deep Equilibrium Canonicalizer](https://mlsanity.com/paper/2508.14187/) (2025)
- [Adaptive Image Zoom-in with Bounding Box Transformation for UAV Object Detection](https://mlsanity.com/paper/2602.07512/) (2026)
- [VPOcc: Exploiting Vanishing Point for 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2408.03551/) (2024)
- [Foveated Instance Segmentation](https://mlsanity.com/paper/2503.21854/) (2025)
- [Cost-Efficient Multi-Scale Fovea for Semantic-Based Visual Search Attention](https://mlsanity.com/paper/2604.03836/) (2026)
- [Telescope: Learnable Hyperbolic Foveation for Ultra-Long-Range Object Detection](https://mlsanity.com/paper/2604.06332/) (2026)
- [UHR-DETR: Efficient End-to-End Small Object Detection for Ultra-High-Resolution Remote Sensing Imagery](https://mlsanity.com/paper/2604.21435/) (2026)
- [DiffVAS: Diffusion-Guided Visual Active Search in Partially Observable Environments](https://mlsanity.com/paper/2605.15519/) (2026)
- [WarpI2I: Image Warping for Image-to-Image Translation](https://mlsanity.com/paper/2606.31018/) (2026)
- [HiResNets: Native Full-HD Video Recognition with Foveal Residual Streams](https://mlsanity.com/paper/2608.02140/) (2026)
- [Variable-Granularity Tokenization for High-Resolution Object Detection](https://mlsanity.com/paper/2608.28706/) (2026)

## Abstract (arXiv)

Efficient processing of high-res video streams is safety-critical for many robotics applications such as autonomous driving. To maintain real-time performance, many practical systems downsample the video stream. But this can hurt downstream tasks such as (small) object detection. Instead, we take inspiration from biological vision systems that allocate more foveal "pixels" to salient parts of the scene. We introduce FOVEA, an approach for intelligent downsampling that ensures salient image regions remain "magnified" in the downsampled output. Given a high-res image, FOVEA applies a differentiable resampling layer that outputs a small fixed-size image canvas, which is then processed with a differentiable vision module (e.g., object detection network), whose output is then differentiably backward mapped onto the original image size. The key idea is to resample such that background pixels can make room for salient pixels of interest. In order to ensure the overall pipeline remains efficient, FOVEA makes use of cheap and readily available cues for saliency, including dataset-specific spatial priors or temporal priors computed from object predictions in the recent past. On the autonomous driving datasets Argoverse-HD and BDD100K, our proposed method boosts the detection AP over standard Faster R-CNN, both with and without finetuning. Without any noticeable increase in compute, we improve accuracy on small objects by over 2x without degrading performance on large objects. Finally, FOVEA sets a new record for streaming AP (from 17.8 to 23.0 on a GTX 1080 Ti GPU), a metric designed to capture both accuracy and latency.