# Answerer in Questioner's Mind: Information Theoretic Approach to Goal-Oriented Visual Dialog

arXiv: https://arxiv.org/abs/1802.03881 · HTML: https://mlsanity.com/paper/1802.03881/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2018.

## Built on by (most cited first, 11 of 11)

- [Open-domain clarification question generation without question examples](https://mlsanity.com/paper/2110.09779/) (2021)
- [Simple Baselines for Interactive Video Retrieval with Questions and Answers](https://mlsanity.com/paper/2308.10402/) (2023)
- [Asking More Informative Questions for Grounded Retrieval](https://mlsanity.com/paper/2311.08584/) (2023)
- [Saying the Unseen: Video Descriptions via Dialog Agents](https://mlsanity.com/paper/2106.14069/) (2021)
- [DRPG (Decompose, Retrieve, Plan, Generate): An Agentic Framework for Academic Rebuttal](https://mlsanity.com/paper/2601.18081/) (2026)
- [What Does The User Want? Information Gain for Hierarchical Dialogue Policy Optimisation](https://mlsanity.com/paper/2109.07129/) (2021)
- [PROGrasp: Pragmatic Human-Robot Communication for Object Grasping](https://mlsanity.com/paper/2309.07759/) (2023)
- [Communicating with Speakers and Listeners of Different Pragmatic Levels](https://mlsanity.com/paper/2410.05851/) (2024)
- [JIR-Arena: The First Benchmark Dataset for Just-in-time Information Recommendation](https://mlsanity.com/paper/2505.13550/) (2025)
- [Dialogue Telemetry: Turn-Level Instrumentation for Autonomous Information Gathering](https://mlsanity.com/paper/2601.09570/) (2026)
- [Aligning Medical Conversational AI through Online Reinforcement Learning with Information-Theoretic Rewards](https://mlsanity.com/paper/2601.17828/) (2026)

## Abstract (arXiv)

Goal-oriented dialog has been given attention due to its numerous applications in artificial intelligence. Goal-oriented dialogue tasks occur when a questioner asks an action-oriented question and an answerer responds with the intent of letting the questioner know a correct action to take. To ask the adequate question, deep learning and reinforcement learning have been recently applied. However, these approaches struggle to find a competent recurrent neural questioner, owing to the complexity of learning a series of sentences. Motivated by theory of mind, we propose "Answerer in Questioner's Mind" (AQM), a novel information theoretic algorithm for goal-oriented dialog. With AQM, a questioner asks and infers based on an approximated probabilistic model of the answerer. The questioner figures out the answerer's intention via selecting a plausible question by explicitly calculating the information gain of the candidate intentions and possible answers to each question. We test our framework on two goal-oriented visual dialog tasks: "MNIST Counting Dialog" and "GuessWhat?!". In our experiments, AQM outperforms comparative algorithms by a large margin.