# Introspective Perception: Learning to Predict Failures in Vision Systems

arXiv: https://arxiv.org/abs/1607.08665 · HTML: https://mlsanity.com/paper/1607.08665/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2016.

## Built on by (most cited first, 17 of 17)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress](https://mlsanity.com/paper/2410.04640/) (2024)
- [A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models](https://mlsanity.com/paper/2502.13187/) (2025)
- [From Machine Learning to Robotics: Challenges and Opportunities for Embodied Intelligence](https://mlsanity.com/paper/2110.15245/) (2021)
- [An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)](https://mlsanity.com/paper/2302.13814/) (2023)
- [Trust Your Robots! Predictive Uncertainty Estimation of Neural Networks with Sparse Gaussian Processes](https://mlsanity.com/paper/2109.09690/) (2021)
- [FSNet: A Failure Detection Framework for Semantic Segmentation](https://mlsanity.com/paper/2108.08748/) (2021)
- [Rule-Based Error Detection and Correction to Operationalize Movement Trajectory Classification](https://mlsanity.com/paper/2308.14250/) (2023)
- [Error Detection and Constraint Recovery in Hierarchical Multi-Label Classification without Prior Knowledge](https://mlsanity.com/paper/2407.15192/) (2024)
- [Towards Disturbance-Free Visual Mobile Manipulation](https://mlsanity.com/paper/2112.12612/) (2021)
- [Diversity Measures: Domain-Independent Proxies for Failure in Language Model Queries](https://mlsanity.com/paper/2308.11189/) (2023)
- [Cost-Sensitive Uncertainty-Based Failure Recognition for Object Detection](https://mlsanity.com/paper/2404.17427/) (2024)
- [CogSense: A Cognitively Inspired Framework for Perception Adaptation](https://mlsanity.com/paper/2107.10456/) (2021)
- [Competence-Aware Path Planning via Introspective Perception](https://mlsanity.com/paper/2109.13974/) (2021)
- [Position: Artificial Intelligence Needs Meta Intelligence -- the Case for Metacognitive AI](https://mlsanity.com/paper/2605.15567/) (2026)
- [Rethinking the Implications of Human Feedback for Preference Learning in Human-Robot Collaboration](https://mlsanity.com/paper/2609.13982/) (2026)
- [Auditing Latent-Space Monitors for Autonomous Driving](https://mlsanity.com/paper/2609.30557/) (2026)

## Abstract (arXiv)

As robots aspire for long-term autonomous operations in complex dynamic environments, the ability to reliably take mission-critical decisions in ambiguous situations becomes critical. This motivates the need to build systems that have situational awareness to assess how qualified they are at that moment to make a decision. We call this self-evaluating capability as introspection. In this paper, we take a small step in this direction and propose a generic framework for introspective behavior in perception systems. Our goal is to learn a model to reliably predict failures in a given system, with respect to a task, directly from input sensor data. We present this in the context of vision-based autonomous MAV flight in outdoor natural environments, and show that it effectively handles uncertain situations.