IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agents
Not on the map. Why?
Abstract (arXiv)
Seamless interaction between AI agents and humans using natural language remains a key goal in AI research. This paper addresses the challenges of developing interactive agents capable of understanding and executing grounded natural language instructions through the IGLU competition at NeurIPS. Despite advancements, challenges such as a scarcity of appropriate datasets and the need for effective evaluation platforms persist. We introduce a scalable data collection tool for gathering interactive grounded language instructions within a Minecraft-like environment, resulting in a Multi-Modal dataset with around 9,000 utterances and over 1,000 clarification questions. Additionally, we present a Human-in-the-Loop interactive evaluation platform for qualitative analysis and comparison of agent performance through multi-turn communication with human annotators. We offer to the community these assets referred to as IDAT (IGLU Dataset And Toolkit) which aim to advance the development of intelligent, interactive AI agents and provide essential resources for further research.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 58 of its 101 references that are in the map. First posted on arXiv 12 Jul 2024.
Builds on
58 of 101 references.
References · 58
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
- 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
- 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
- 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
- 2020Recipes for building an open-domain chatbotcited by 593
- 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
38 more
- 2020Towards a Human-like Open-Domain Chatbotcited by 467
- 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
- 2019Vision-and-Dialog Navigationcited by 246
- 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
- 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
- 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
- 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
- 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
- 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
- 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
- 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
- 2019Executing Instructions in Situated Collaborative Interactionscited by 68
- 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
- 2020ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)cited by 55
- 2020Imitating Interactive Intelligencecited by 55
- 2020Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedbackcited by 44
- 2016Learning Language Games through Interactioncited by 43
- 2015Program Synthesis using Natural Languagecited by 41
- 2019Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Studycited by 40
- 2021Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learningcited by 40
- 2023Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedbackcited by 31
- 2019CraftAssist: A Framework for Dialogue-enabled Interactive Agentscited by 27
- 2017Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Modelscited by 26
- 2020An Imitation Game for Learning Semantic Parsers from User Interactioncited by 23
- 2022Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021cited by 23
- 2017Iris: A Conversational Agent for Complex Taskscited by 22
- 2018Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learningcited by 21
- 2017Naturalizing a Programming Language via Interactive Learningcited by 20
- 2018Adversarial Learning of Task-Oriented Neural Dialog Modelscited by 17
- 2024Assessing and Verifying Task Utility in LLM-Powered Applicationscited by 15
- 2019Why Build an Assistant in Minecraft?cited by 14
- 2023Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructionscited by 13
- 2021A Framework for Learning to Request Rich and Contextually Useful Information from Humanscited by 12
- 2020Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systemscited by 10
- 2022Collecting Interactive Multi-modal Datasets for Grounded Language Understandingcited by 10
- 2019Improving Natural Language Interaction with Robots Using Advicecited by 9
- 2022IGLU Gridworld: Simple and Fast Environment for Embodied Dialog Agentscited by 9
- 2022Learning to Solve Voxel Building Embodied Tasks from Pixels and Natural Language Instructionscited by 9
50 most cited of 58 All 58 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.