# Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedback

arXiv: https://arxiv.org/abs/2304.10750 · HTML: https://mlsanity.com/paper/2304.10750/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 59 references that are in the map.
- First posted on arXiv 21 Apr 2023.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [REFINER: Reasoning Feedback on Intermediate Representations](https://mlsanity.com/paper/2304.01904/) (2023)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions](https://mlsanity.com/paper/2109.05794/) (2021)
- [ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)](https://mlsanity.com/paper/2009.11352/) (2020)
- [Imitating Interactive Intelligence](https://mlsanity.com/paper/2012.05672/) (2020)
- [Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedback](https://mlsanity.com/paper/2112.09737/) (2021)
- [NL-EDIT: Correcting semantic parse errors through natural language interaction](https://mlsanity.com/paper/2103.14540/) (2021)
- [Learning to Execute Actions or Ask Clarification Questions](https://mlsanity.com/paper/2204.08373/) (2022)
- [CraftAssist: A Framework for Dialogue-enabled Interactive Agents](https://mlsanity.com/paper/1907.08584/) (2019)
- [The MineRL BASALT Competition on Learning from Human Feedback](https://mlsanity.com/paper/2107.01969/) (2021)
- [Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021](https://mlsanity.com/paper/2205.02388/) (2022)
- [Open-domain clarification question generation without question examples](https://mlsanity.com/paper/2110.09779/) (2021)
- [NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment](https://mlsanity.com/paper/2110.06536/) (2021)
- [IGLU 2022: Interactive Grounded Language Understanding in a Collaborative Environment at NeurIPS 2022](https://mlsanity.com/paper/2205.13771/) (2022)
- [Collecting Interactive Multi-modal Datasets for Grounded Language Understanding](https://mlsanity.com/paper/2211.06552/) (2022)
- [Improving Natural Language Interaction with Robots Using Advice](https://mlsanity.com/paper/1905.04655/) (2019)
- [IGLU Gridworld: Simple and Fast Environment for Embodied Dialog Agents](https://mlsanity.com/paper/2206.00142/) (2022)
- [Learning to Solve Voxel Building Embodied Tasks from Pixels and Natural Language Instructions](https://mlsanity.com/paper/2211.00688/) (2022)
- [Unsupervised Question Clarity Prediction Through Retrieved Item Coherency](https://mlsanity.com/paper/2208.04882/) (2022)
- [Improving Neural Model Performance through Natural Language Feedback on Their Explanations](https://mlsanity.com/paper/2104.08765/) (2021)
- [Deciding Whether to Ask Clarifying Questions in Large-Scale Spoken Language Understanding](https://mlsanity.com/paper/2109.12451/) (2021)

## Built on by (most cited first, 30 of 32)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects](https://mlsanity.com/paper/2401.03428/) (2024)
- [AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathways](https://mlsanity.com/paper/2406.02630/) (2024)
- [A Survey on Code Generation with LLM-based Agents](https://mlsanity.com/paper/2508.00083/) (2025)
- [MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow](https://mlsanity.com/paper/2503.18968/) (2025)
- [Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents](https://mlsanity.com/paper/2311.11797/) (2023)
- [Large Language Model-based Human-Agent Collaboration for Complex Task Solving](https://mlsanity.com/paper/2402.12914/) (2024)
- [GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modeling](https://mlsanity.com/paper/2505.19234/) (2025)
- [Interactive Natural Language Processing](https://mlsanity.com/paper/2305.13246/) (2023)
- [LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey](https://mlsanity.com/paper/2505.00753/) (2025)
- [SurgRAW: Multi-Agent Workflow with Chain of Thought Reasoning for Robotic Surgical Video Analysis](https://mlsanity.com/paper/2503.10265/) (2025)
- [GuideBench: Benchmarking Domain-Oriented Guideline Following for LLM Agents](https://mlsanity.com/paper/2505.11368/) (2025)
- [Transforming Human-Centered AI Collaboration: Redefining Embodied Agents Capabilities through Interactive Grounded Language Instructions](https://mlsanity.com/paper/2305.10783/) (2023)
- [Computational Experiments Meet Large Language Model Based Agents: A Survey and Perspective](https://mlsanity.com/paper/2402.00262/) (2024)
- [Causal Agent based on Large Language Model](https://mlsanity.com/paper/2408.06849/) (2024)
- [AssistanceZero: Scalably Solving Assistance Games](https://mlsanity.com/paper/2504.07091/) (2025)
- [Text2BIM: Generating Building Models Using a Large Language Model-based Multi-Agent Framework](https://mlsanity.com/paper/2408.08054/) (2024)
- [A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomy](https://mlsanity.com/paper/2506.09420/) (2025)
- [Large Language Models as Minecraft Agents](https://mlsanity.com/paper/2402.08392/) (2024)
- [Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use](https://mlsanity.com/paper/2410.24218/) (2024)
- [Towards a copilot in BIM authoring tool using a large language model-based agent for intelligent human-machine interaction](https://mlsanity.com/paper/2406.16903/) (2024)
- [How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyond](https://mlsanity.com/paper/2501.05714/) (2025)
- [MotiveBench: How Far Are We From Human-Like Motivational Reasoning in Large Language Models?](https://mlsanity.com/paper/2506.13065/) (2025)
- [Preventing Rogue Agents Improves Multi-Agent Collaboration](https://mlsanity.com/paper/2502.05986/) (2025)
- [Carbon and Silicon, Coexist or Compete? A Survey on Human-AI Interactions in Agent-based Modeling and Simulation](https://mlsanity.com/paper/2502.18145/) (2025)
- [Intelli-Planner: Towards Customized Urban Planning via Large Language Model Empowered Reinforcement Learning](https://mlsanity.com/paper/2601.21212/) (2026)
- [Divide by Question, Conquer by Agent: SPLIT-RAG with Question-Driven Graph Partitioning](https://mlsanity.com/paper/2505.13994/) (2025)
- [IDAT: A Multi-Modal Dataset and Toolkit for Building and Evaluating Interactive Task-Solving Agents](https://mlsanity.com/paper/2407.08898/) (2024)
- [Before Agents Speak: Pre-hoc Failure Risk Inference in Multi-Agent Systems](https://mlsanity.com/paper/2607.26836/) (2026)

## Abstract (arXiv)

Many approaches to Natural Language Processing (NLP) tasks often treat them as single-step problems, where an agent receives an instruction, executes it, and is evaluated based on the final outcome. However, human language is inherently interactive, as evidenced by the back-and-forth nature of human conversations. In light of this, we posit that human-AI collaboration should also be interactive, with humans monitoring the work of AI agents and providing feedback that the agent can understand and utilize. Further, the AI agent should be able to detect when it needs additional information and proactively ask for help. Enabling this scenario would lead to more natural, efficient, and engaging human-AI collaborations. In this work, we explore these directions using the challenging task defined by the IGLU competition, an interactive grounded language understanding task in a MineCraft-like world. We explore multiple types of help players can give to the AI to guide it and analyze the impact of this help in AI behavior, resulting in performance improvements.