# Learning to Execute Actions or Ask Clarification Questions

arXiv: https://arxiv.org/abs/2204.08373 · HTML: https://mlsanity.com/paper/2204.08373/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 18 Apr 2022.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Vision-and-Dialog Navigation](https://mlsanity.com/paper/1907.04957/) (2019)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1905.08743/) (2019)
- [MineRL: A Large-Scale Dataset of Minecraft Demonstrations](https://mlsanity.com/paper/1907.13440/) (2019)
- [GuessWhat?! Visual object discovery through multi-modal dialogue](https://mlsanity.com/paper/1611.08481/) (2016)
- [Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning](https://mlsanity.com/paper/1909.01871/) (2019)
- [Mapping Instructions and Visual Observations to Actions with Reinforcement Learning](https://mlsanity.com/paper/1704.08795/) (2017)
- [Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learning](https://mlsanity.com/paper/1712.07294/) (2017)
- [Executing Instructions in Situated Collaborative Interactions](https://mlsanity.com/paper/1910.03655/) (2019)
- [Just Ask:An Interactive Learning Framework for Vision and Language Navigation](https://mlsanity.com/paper/1912.00915/) (2019)
- [A Sequence-to-Sequence Approach to Dialogue State Tracking](https://mlsanity.com/paper/2011.09553/) (2020)
- [Span-ConveRT: Few-shot Span Extraction for Dialog with Pretrained Conversational Representations](https://mlsanity.com/paper/2005.08866/) (2020)
- [RMM: A Recursive Mental Model for Dialog Navigation](https://mlsanity.com/paper/2005.00728/) (2020)
- [Learning by Asking Questions](https://mlsanity.com/paper/1712.01238/) (2017)
- [Representation Learning for Grounded Spatial Reasoning](https://mlsanity.com/paper/1707.03938/) (2017)
- [Dynamic Schema Graph Fusion Network for Multi-Domain Dialogue State Tracking](https://mlsanity.com/paper/2204.06677/) (2022)
- [Why Build an Assistant in Minecraft?](https://mlsanity.com/paper/1907.09273/) (2019)
- [Situated Mapping of Sequential Instructions to Actions with Single-step Reward Observation](https://mlsanity.com/paper/1805.10209/) (2018)
- [Robust and Interpretable Grounding of Spatial References with Relation Networks](https://mlsanity.com/paper/2005.00696/) (2020)
- [Leveraging Review Properties for Effective Recommendation](https://mlsanity.com/paper/2102.03089/) (2021)
- [Negative Confidence-Aware Weakly Supervised Binary Classification for Effective Review Helpfulness Classification](https://mlsanity.com/paper/2008.06487/) (2020)

## Built on by (most cited first, 30 of 30)

- [Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agents](https://mlsanity.com/paper/2402.09205/) (2024)
- [Instruction Tuning With Loss Over Instructions](https://mlsanity.com/paper/2405.14394/) (2024)
- [DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning](https://mlsanity.com/paper/2309.05173/) (2023)
- [Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedback](https://mlsanity.com/paper/2304.10750/) (2023)
- [Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions](https://mlsanity.com/paper/2402.06509/) (2024)
- [Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner](https://mlsanity.com/paper/2305.01711/) (2023)
- [A Survey of Conversational Search](https://mlsanity.com/paper/2410.15576/) (2024)
- [Rethinking Semi-supervised Learning with Language Models](https://mlsanity.com/paper/2305.13002/) (2023)
- [Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset](https://mlsanity.com/paper/2302.14406/) (2023)
- [ArchCode: Incorporating Software Requirements in Code Generation with Large Language Models](https://mlsanity.com/paper/2408.00994/) (2024)
- [RAcQUEt: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMs](https://mlsanity.com/paper/2412.13835/) (2024)
- [Large Language Models as Minecraft Agents](https://mlsanity.com/paper/2402.08392/) (2024)
- [When and What to Ask Through World States and Text Instructions: IGLU NLP Challenge Solution](https://mlsanity.com/paper/2305.05754/) (2023)
- [R2H: Building Multimodal Navigation Helpers that Respond to Help Requests](https://mlsanity.com/paper/2305.14260/) (2023)
- [Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requests](https://mlsanity.com/paper/2401.17039/) (2024)
- [Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft](https://mlsanity.com/paper/2406.17553/) (2024)
- [Self Contrastive Learning for Session-based Recommendation](https://mlsanity.com/paper/2306.01266/) (2023)
- ["Are you telling me to put glasses on the dog?'' Content-Grounded Annotation of Instruction Clarification Requests in the CoDraw Dataset](https://mlsanity.com/paper/2306.02377/) (2023)
- [Towards a Neural Era in Dialogue Management for Collaboration: A Literature Survey](https://mlsanity.com/paper/2307.09021/) (2023)
- ['What are you referring to?' Evaluating the Ability of Multi-Modal Dialogue Models to Process Clarificational Exchanges](https://mlsanity.com/paper/2307.15554/) (2023)
- [Nebula: A discourse aware Minecraft Builder](https://mlsanity.com/paper/2406.18164/) (2024)
- [Repairs in a Block World: A New Benchmark for Handling User Corrections with Multi-Modal Language Models](https://mlsanity.com/paper/2409.14247/) (2024)
- [It Depends: Resolving Referential Ambiguity in Minimal Contexts with Commonsense Knowledge](https://mlsanity.com/paper/2509.16107/) (2025)
- [Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysis](https://mlsanity.com/paper/2306.07664/) (2023)
- [It Couldn't Help But Overhear: On the Limits of Modelling Meta-Communicative Grounding Acts with Supervised Learning](https://mlsanity.com/paper/2405.01139/) (2024)
- [Attention-based Ingredient Phrase Parser](https://mlsanity.com/paper/2210.02535/) (2022)
- [Solving Dialogue Grounding Embodied Task in a Simulated Environment using Further Masked Language Modeling](https://mlsanity.com/paper/2306.12387/) (2023)
- [Lexical Entrainment for Conversational Systems](https://mlsanity.com/paper/2310.09651/) (2023)
- [Improving Agent Interactions in Virtual Environments with Language Models](https://mlsanity.com/paper/2402.05440/) (2024)
- [Beyond Local Surprise: Grounded Dialogue as Selective Belief Revision under Referential Uncertainty](https://mlsanity.com/paper/2608.26035/) (2026)

## Abstract (arXiv)

Collaborative tasks are ubiquitous activities where a form of communication is required in order to reach a joint goal. Collaborative building is one of such tasks. We wish to develop an intelligent builder agent in a simulated building environment (Minecraft) that can build whatever users wish to build by just talking to the agent. In order to achieve this goal, such agents need to be able to take the initiative by asking clarification questions when further information is needed. Existing works on Minecraft Corpus Dataset only learn to execute instructions neglecting the importance of asking for clarifications. In this paper, we extend the Minecraft Corpus Dataset by annotating all builder utterances into eight types, including clarification questions, and propose a new builder agent model capable of determining when to ask or execute instructions. Experimental results show that our model achieves state-of-the-art performance on the collaborative building task with a substantial improvement. We also define two new tasks, the learning to ask task and the joint learning task. The latter consists of solving both collaborating building and learning to ask tasks jointly.