# Evaluating and Improving Interactions with Hazy Oracles

arXiv: https://arxiv.org/abs/2110.10206 · HTML: https://mlsanity.com/paper/2110.10206/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 50 references that are in the map.
- First posted on arXiv 19 Oct 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [Consistent Estimators for Learning to Defer to an Expert](https://mlsanity.com/paper/2006.01862/) (2020)
- [A Novel Performance Evaluation Methodology for Single-Target Trackers](https://mlsanity.com/paper/1503.01313/) (2015)
- [Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers](https://mlsanity.com/paper/1805.08206/) (2018)
- [BayesOD: A Bayesian Approach for Uncertainty Estimation in Deep Object Detectors](https://mlsanity.com/paper/1903.03838/) (2019)
- [Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions](https://mlsanity.com/paper/1710.06280/) (2017)
- [Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction](https://mlsanity.com/paper/1806.03831/) (2018)
- [Interactive Full Image Segmentation by Considering All Regions Jointly](https://mlsanity.com/paper/1812.01888/) (2018)
- [Why Does a Visual Question Have Different Answers?](https://mlsanity.com/paper/1908.04342/) (2019)
- [Composing Pick-and-Place Tasks By Grounding Language](https://mlsanity.com/paper/2102.08094/) (2021)
- [The Promise of Premise: Harnessing Question Premises in Visual Question Answering](https://mlsanity.com/paper/1705.00601/) (2017)
- [Learning Intelligent Dialogs for Bounding Box Annotation](https://mlsanity.com/paper/1712.08087/) (2017)
- [Click Carving: Segmenting Objects in Video with Point Clicks](https://mlsanity.com/paper/1607.01115/) (2016)
- [Leveraging Uncertainty in Deep Learning for Selective Classification](https://mlsanity.com/paper/1905.09509/) (2019)
- [Click Here: Human-Localized Keypoints as Guidance for Viewpoint Estimation](https://mlsanity.com/paper/1703.09859/) (2017)
- [Panoptic Image Annotation with a Collaborative Assistant](https://mlsanity.com/paper/1906.06798/) (2019)
- [Ground-truth or DAER: Selective Re-query of Secondary Information](https://mlsanity.com/paper/2009.07414/) (2020)

## Built on by (most cited first, 1 of 1)

- [From Control to Foresight: Simulation as a New Paradigm for Human-Agent Collaboration](https://mlsanity.com/paper/2603.11677/) (2026)

## Abstract (arXiv)

Many AI systems integrate sensor inputs, world knowledge, and human-provided information to perform inference. While such systems often treat the human input as flawless, humans are better thought of as hazy oracles whose input may be ambiguous or outside of the AI system's understanding. In such situations it makes sense for the AI system to defer its inference while it disambiguates the human-provided information by, for example, asking the human to rephrase the query. Though this approach has been considered in the past, current work is typically limited to application-specific methods and non-standardized human experiments. We instead introduce and formalize a general notion of deferred inference. Using this formulation, we then propose a novel evaluation centered around the Deferred Error Volume (DEV) metric, which explicitly considers the tradeoff between error reduction and the additional human effort required to achieve it. We demonstrate this new formalization and an innovative deferred inference method on the disparate tasks of Single-Target Video Object Tracking and Referring Expression Comprehension, ultimately reducing error by up to 48% without any change to the underlying model or its parameters.