mlsanityData to 7 Oct 2026

Collecting Interactive Multi-modal Datasets for Grounded Language Understanding

arXiv 2211.06552 ↗First posted 12 Nov 2022cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Human intelligence can remarkably adapt quickly to new tasks and environments. Starting from a very young age, humans acquire new skills and learn how to solve new tasks either by imitating the behavior of others or by following provided natural language instructions. To facilitate research which can enable similar capabilities in machines, we made the following contributions (1) formalized the collaborative embodied agent using natural language task; (2) developed a tool for extensive and scalable data collection; and (3) collected the first dataset for interactive grounded language understanding.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 12 of its 34 references that are in the map. First posted on arXiv 12 Nov 2022.

Builds on

12 of 34 references.

References · 12

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Asking Clarifying Questions in Open-Domain Information-Seeking Conversationscited by 136
  3. 2018Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Informationcited by 110
  4. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  5. 2020ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)cited by 55
  6. 2020Imitating Interactive Intelligencecited by 55
  7. 2020Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learningcited by 32
  8. 2022Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021cited by 23
  9. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  10. 2021NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environmentcited by 15
  11. 2022IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022cited by 14
  12. 2022Unsupervised Question Clarity Prediction Through Retrieved Item Coherencycited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 0; 2026: 1.

10 papers.

  1. 2024Asking the Right Question at the Right Time: Human and Model Uncertainty Guidance to Ask Clarification Questionscited by 32
  2. 2023Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedbackcited by 31
  3. 2024S-Agents: Self-organizing Agents in Open-ended Environmentscited by 22
  4. 2023Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructionscited by 13
  5. 2023Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Datasetcited by 11
  6. 2024Large Language Models as Minecraft Agentscited by 7
  7. 2024Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requestscited by 4
  8. 2024A LLM Benchmark based on the Minecraft Builder Dialog Agent Taskcited by 4
  9. 2024IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agentscited by 1
  10. 2026When Contextual Inference Fails: Cancelability in Interactive Instruction Followingcited by 0