# Collecting Interactive Multi-modal Datasets for Grounded Language Understanding

arXiv: https://arxiv.org/abs/2211.06552 · HTML: https://mlsanity.com/paper/2211.06552/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 34 references that are in the map.
- First posted on arXiv 12 Nov 2022.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Asking Clarifying Questions in Open-Domain Information-Seeking Conversations](https://mlsanity.com/paper/1907.06554/) (2019)
- [Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information](https://mlsanity.com/paper/1805.04655/) (2018)
- [Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions](https://mlsanity.com/paper/2109.05794/) (2021)
- [ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)](https://mlsanity.com/paper/2009.11352/) (2020)
- [Imitating Interactive Intelligence](https://mlsanity.com/paper/2012.05672/) (2020)
- [Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning](https://mlsanity.com/paper/2011.00517/) (2020)
- [Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021](https://mlsanity.com/paper/2205.02388/) (2022)
- [Continual Learning for Grounded Instruction Generation by Observing Human Following Behavior](https://mlsanity.com/paper/2108.04812/) (2021)
- [NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment](https://mlsanity.com/paper/2110.06536/) (2021)
- [IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022](https://mlsanity.com/paper/2205.13771/) (2022)
- [Unsupervised Question Clarity Prediction Through Retrieved Item Coherency](https://mlsanity.com/paper/2208.04882/) (2022)

## Built on by (most cited first, 10 of 10)

- [Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedback](https://mlsanity.com/paper/2304.10750/) (2023)
- [Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questions](https://mlsanity.com/paper/2402.06509/) (2024)
- [S-Agents: Self-organizing Agents in Open-ended Environments](https://mlsanity.com/paper/2402.04578/) (2024)
- [Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions](https://mlsanity.com/paper/2305.10783/) (2023)
- [Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset](https://mlsanity.com/paper/2302.14406/) (2023)
- [Large Language Models as Minecraft Agents](https://mlsanity.com/paper/2402.08392/) (2024)
- [Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requests](https://mlsanity.com/paper/2401.17039/) (2024)
- [A LLM Benchmark based on the Minecraft Builder Dialog Agent Task](https://mlsanity.com/paper/2407.12734/) (2024)
- [IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agents](https://mlsanity.com/paper/2407.08898/) (2024)
- [When Contextual Inference Fails: Cancelability in Interactive Instruction Following](https://mlsanity.com/paper/2603.19997/) (2026)

## Abstract (arXiv)

Human intelligence can remarkably adapt quickly to new tasks and environments. Starting from a very young age, humans acquire new skills and learn how to solve new tasks either by imitating the behavior of others or by following provided natural language instructions. To facilitate research which can enable similar capabilities in machines, we made the following contributions (1) formalized the collaborative embodied agent using natural language task; (2) developed a tool for extensive and scalable data collection; and (3) collected the first dataset for interactive grounded language understanding.