# Policy-based Foveated Imaging and Perception

arXiv: https://arxiv.org/abs/2606.02565 · HTML: https://mlsanity.com/paper/2606.02565/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 54 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Builds on (most cited first, 27 of 27)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [ByteTrack: Multi-Object Tracking by Associating Every Detection Box](https://mlsanity.com/paper/2110.06864/) (2021)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [YOLOv11: An Overview of the Key Architectural Enhancements](https://mlsanity.com/paper/2410.17725/) (2024)
- [Recurrent Models of Visual Attention](https://mlsanity.com/paper/1406.6247/) (2014)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [Distractor-aware Siamese Networks for Visual Object Tracking](https://mlsanity.com/paper/1808.06048/) (2018)
- [TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text](https://mlsanity.com/paper/2105.05486/) (2021)
- [Hierarchical Imitation and Reinforcement Learning](https://mlsanity.com/paper/1803.00590/) (2018)
- [SoccerNet-Tracking: Multiple Object Tracking Dataset and Benchmark in Soccer Videos](https://mlsanity.com/paper/2204.06918/) (2022)
- [FoV-NeRF: Foveated Neural Radiance Fields for Virtual Reality](https://mlsanity.com/paper/2103.16365/) (2021)
- [MixFormerV2: Efficient Fully Transformer Tracking](https://mlsanity.com/paper/2305.15896/) (2023)
- [RoadText-1K: Text Detection & Recognition Dataset for Driving Videos](https://mlsanity.com/paper/2005.09496/) (2020)
- [Eye, Robot: Learning to Look to Act with a BC-RL Perception-Action Loop](https://mlsanity.com/paper/2506.10968/) (2025)
- [Recurrent Attention Models for Depth-Based Person Identification](https://mlsanity.com/paper/1611.07212/) (2016)
- [NSNet: Non-saliency Suppression Sampler for Efficient Video Recognition](https://mlsanity.com/paper/2207.10388/) (2022)
- [Emulating Human-like Adaptive Vision for Efficient and Flexible Machine Visual Perception](https://mlsanity.com/paper/2509.15333/) (2025)
- [Object Detection Through Exploration With A Foveated Visual Field](https://mlsanity.com/paper/1408.0814/) (2014)
- [Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing](https://mlsanity.com/paper/2603.12254/) (2026)
- [ICDAR 2015 Text Reading in the Wild Competition](https://mlsanity.com/paper/1506.03184/) (2015)
- [Foveation in the Era of Deep Learning](https://mlsanity.com/paper/2312.01450/) (2023)
- [Foveated Rendering: a State-of-the-Art Survey](https://mlsanity.com/paper/2211.07969/) (2022)
- [Towards Attention-aware Foveated Rendering](https://mlsanity.com/paper/2302.01368/) (2023)
- [Human eye inspired log-polar pre-processing for neural networks](https://mlsanity.com/paper/1911.01141/) (2019)

## Built on by (most cited first, 1 of 1)

- [FoveaVLA: Foveation Actions Replace the Wrist Camera in VLAs](https://mlsanity.com/paper/2610.20064/) (2026)

## Abstract (arXiv)

Ultra-high-resolution image sensors offer the potential to capture fine spatial details critical for many visual perception tasks, but acquiring and processing all pixels at full resolution is often infeasible under realistic bandwidth, latency, and power constraints. Existing approaches address this challenge through acquisition strategies such as spatial or temporal downsampling, which irrevocably discard information before task relevance can be assessed. In this work, we introduce a real-time, predictive, and task-aware foveated imaging system that operates directly at image acquisition time. Leveraging emerging dual-stream sensor architectures, our method dynamically allocates limited pixel bandwidth to task-relevant regions of interest while maintaining a low-resolution global context. We formulate foveated acquisition as a sensor attention policy-learning problem, in which past observations guide actions that determine future measurements, closing the perception-acquisition loop. Through extensive simulation across multiple perception tasks, we demonstrate that our approach achieves high task performance under strict pixel budgets and significantly outperforms relevant baselines operating at the same bandwidth. We further validate our system on a 200-megapixel dual-stream sensor, capturing real-world videos under realistic bandwidth and latency constraints, demonstrating the practical feasibility of task-driven, acquisition-time foveated imaging.