# A Dataset for Developing and Benchmarking Active Vision

arXiv: https://arxiv.org/abs/1702.08272 · HTML: https://mlsanity.com/paper/1702.08272/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2017.

## Built on by (most cited first, 30 of 50)

- [Unconstrained Scene Generation with Locally Conditioned Radiance Fields](https://mlsanity.com/paper/2104.00670/) (2021)
- [Vision-Language Navigation with Random Environmental Mixup](https://mlsanity.com/paper/2106.07876/) (2021)
- [SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency](https://mlsanity.com/paper/2112.01001/) (2021)
- [SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction from Video Data](https://mlsanity.com/paper/2105.08612/) (2021)
- [World Models and Predictive Coding for Cognitive and Developmental Robotics: Frontiers and Challenges](https://mlsanity.com/paper/2301.05832/) (2023)
- [A Survey on Human-aware Robot Navigation](https://mlsanity.com/paper/2106.11650/) (2021)
- [Interactron: Embodied Adaptive Object Detection](https://mlsanity.com/paper/2202.00660/) (2022)
- [Learning Active Camera for Multi-Object Navigation](https://mlsanity.com/paper/2210.07505/) (2022)
- [Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents](https://mlsanity.com/paper/2110.05769/) (2021)
- [DALL-E for Detection: Language-driven Compositional Image Synthesis for Object Detection](https://mlsanity.com/paper/2206.09592/) (2022)
- [Evidential Active Recognition: Intelligent and Prudent Open-World Embodied Perception](https://mlsanity.com/paper/2311.13793/) (2023)
- [A Survey on RGB-D Datasets](https://mlsanity.com/paper/2201.05761/) (2022)
- [Learning to View: Decision Transformers for Active Object Detection](https://mlsanity.com/paper/2301.09544/) (2023)
- [ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models](https://mlsanity.com/paper/2410.04659/) (2024)
- [Diffusion Models in 3D Vision: A Survey](https://mlsanity.com/paper/2410.04738/) (2024)
- [Beyond Generation: Harnessing Text to Image Models for Object Detection and Segmentation](https://mlsanity.com/paper/2309.05956/) (2023)
- [From Screens to Scenes: A Survey of Embodied AI in Healthcare](https://mlsanity.com/paper/2501.07468/) (2025)
- [Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging](https://mlsanity.com/paper/2103.16693/) (2021)
- [Uncertainty Aware Proposal Segmentation for Unknown Object Detection](https://mlsanity.com/paper/2111.12866/) (2021)
- [Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos](https://mlsanity.com/paper/2411.08753/) (2024)
- [MARS: Multimodal Active Robotic Sensing for Articulated Characterization](https://mlsanity.com/paper/2407.01191/) (2024)
- [Towards real-world navigation with deep differentiable planners](https://mlsanity.com/paper/2108.05713/) (2021)
- [Beyond Visual Field of View: Perceiving 3D Environment with Echoes and Vision](https://mlsanity.com/paper/2207.01136/) (2022)
- [Privacy Risks in Reinforcement Learning for Household Robots](https://mlsanity.com/paper/2306.09273/) (2023)
- [A High-Resolution Dataset for Instance Detection with Multi-View Instance Capture](https://mlsanity.com/paper/2310.19257/) (2023)
- [Self-Annotated 3D Geometric Learning for Smeared Points Removal](https://mlsanity.com/paper/2311.09029/) (2023)
- [Frequency-based Matcher for Long-tailed Semantic Segmentation](https://mlsanity.com/paper/2406.03917/) (2024)
- [Tactile MNIST: Benchmarking Active Tactile Perception](https://mlsanity.com/paper/2506.06361/) (2025)
- [Visual Attention in Imaginative Agents](https://mlsanity.com/paper/2104.00177/) (2021)
- [Object Pose Estimation using Mid-level Visual Representations](https://mlsanity.com/paper/2203.01449/) (2022)

## Abstract (arXiv)

We present a new public dataset with a focus on simulating robotic vision tasks in everyday indoor environments using real imagery. The dataset includes 20,000+ RGB-D images and 50,000+ 2D bounding boxes of object instances densely captured in 9 unique scenes. We train a fast object category detector for instance detection on our data. Using the dataset we show that, although increasingly accurate and fast, the state of the art for object detection is still severely impacted by object scale, occlusion, and viewing direction all of which matter for robotics applications. We next validate the dataset for simulating active vision, and use the dataset to develop and evaluate a deep-network-based system for next best move prediction for object classification using reinforcement learning. Our dataset is available for download at cs.unc.edu/~ammirato/active_vision_dataset_website/.