mlsanityData to 7 Oct 2026

Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions

arXiv 2305.10783 ↗First posted 18 May 2023cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Human intelligence's adaptability is remarkable, allowing us to adjust to new tasks and multi-modal environments swiftly. This skill is evident from a young age as we acquire new abilities and solve problems by imitating others or following natural language instructions. The research community is actively pursuing the development of interactive "embodied agents" that can engage in natural conversations with humans and assist them with real-world tasks. These agents must possess the ability to promptly request feedback in case communication breaks down or instructions are unclear. Additionally, they must demonstrate proficiency in learning new vocabulary specific to a given domain. In this paper, we made the following contributions: (1) a crowd-sourcing tool for collecting grounded language instructions; (2) the largest dataset of grounded language instructions; and (3) several state-of-the-art baselines. These contributions are suitable as a foundation for further research.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 46 of its 77 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

46 of 77 references.

References · 46

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  11. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  12. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
34 more
  1. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  2. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  3. 2020Recipes for building an open-domain chatbotcited by 593
  4. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  5. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  6. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 178
  7. 2021Optimizing Dense Retrieval Model Training with Hard Negativescited by 154
  8. 2019Asking Clarifying Questions in Open-Domain Information-Seeking Conversationscited by 136
  9. 2021Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipelinecited by 65
  10. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  11. 2021MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Taskscited by 62
  12. 2020ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)cited by 55
  13. 2020Imitating Interactive Intelligencecited by 55
  14. 2020Speak to your Parser: Interactive Text-to-SQL with Natural Language Feedbackcited by 44
  15. 2015Program Synthesis using Natural Languagecited by 41
  16. 2019Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Studycited by 40
  17. 2023Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedbackcited by 31
  18. 2019CraftAssist: A Framework for Dialogue-enabled Interactive Agentscited by 27
  19. 2017Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Modelscited by 26
  20. 2020An Imitation Game for Learning Semantic Parsers from User Interactioncited by 23
  21. 2022Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021cited by 23
  22. 2017Iris: A Conversational Agent for Complex Taskscited by 22
  23. 2018Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learningcited by 21
  24. 2018Adversarial Learning of Task-Oriented Neural Dialog Modelscited by 17
  25. 2021NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environmentcited by 15
  26. 2019Why Build an Assistant in Minecraft?cited by 14
  27. 2022IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022cited by 14
  28. 2020Rethinking Supervised Learning and Reinforcement Learning in Task-Oriented Dialogue Systemscited by 10
  29. 2022Collecting Interactive Multi-modal Datasets for Grounded Language Understandingcited by 10
  30. 2022IGLU Gridworld: Simple and Fast Environment for Embodied Dialog Agentscited by 9
  31. 2022Learning to Solve Voxel Building Embodied Tasks from Pixels and Natural Language Instructionscited by 9
  32. 2022Unsupervised Question Clarity Prediction Through Retrieved Item Coherencycited by 8
  33. 2023When and What to Ask Through World States and Text Instructions: IGLU NLP Challenge Solutioncited by 5
  34. 2021Improving Response Quality with Backward Reasoning in Open-domain Dialogue Systemscited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 8; 2025: 4.

13 papers.

Sort:

By area:

  1. 2023Grounding Gaps in Language Model Generationscited by 45
  2. 2025Navigating Rifts in Human-LLM Grounding: Study and Benchmarkcited by 27
  3. 2024S-Agents: Self-organizing Agents in Open-ended Environmentscited by 22
  4. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  5. 2024Large Language Models as Minecraft Agentscited by 7
  6. 2024Taking Action Towards Graceful Interaction: The Effects of Performing Actions on Modelling Policies for Instruction Clarification Requestscited by 4
  7. 2024Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraftcited by 4
  8. 2024In-Memory Learning: A Declarative Learning Framework for Large Language Modelscited by 3
  9. 2024Nebula: A discourse aware Minecraft Buildercited by 3
  10. 2025Carbon and Silicon, Coexist or Compete? A Survey on Human-AI Interactions in Agent-based Modeling and Simulationcited by 3
  11. 2025Gricean Norms as a Basis for Effective Collaborationcited by 3
  12. 2024It Couldn't Help But Overhear: On the Limits of Modelling Meta-Communicative Grounding Acts with Supervised Learningcited by 1
1 more
  1. 2024IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agentscited by 1