Improving Agent Interactions in Virtual Environments with Language Models
On the map: LLM agents and tool use → Reinforcement learning for LLM agents
Abstract (arXiv)
Enhancing AI systems with efficient communication skills for effective human assistance necessitates proactive initiatives from the system side to discern specific circumstances and interact aptly. This research focuses on a collective building assignment in the Minecraft dataset, employing language modeling to enhance task understanding through state-of-the-art methods. These models focus on grounding multi-modal understanding and task-oriented dialogue comprehension tasks, providing insights into their interpretative and responsive capabilities. Our experimental results showcase a substantial improvement over existing methods, indicating a promising direction for future research in this domain.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 30 references that are in the map. First posted on arXiv 8 Feb 2024.
Builds on
23 of 30 references.
References · 23
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
- 2019Publicly Available Clinical BERT Embeddingscited by 995
- 2019Vision-and-Dialog Navigationcited by 246
- 2018Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environmentscited by 238
- 2016GuessWhat?! Visual object discovery through multi-modal dialoguecited by 112
- 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
- 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
11 more
- 2019Executing Instructions in Situated Collaborative Interactionscited by 68
- 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Textscited by 65
- 2019Just Ask:An Interactive Learning Framework for Vision and Language Navigationcited by 53
- 2020RMM: A Recursive Mental Model for Dialog Navigationcited by 32
- 2022Learning to Execute Actions or Ask Clarification Questionscited by 30
- 2017Representation Learning for Grounded Spatial Reasoningcited by 23
- 2023Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learnercited by 23
- 2023Rethinking Semi-supervised Learning with Language Modelscited by 17
- 2018Situated Mapping of Sequential Instructions to Actions with Single-step Reward Observationcited by 12
- 2020Robust and Interpretable Grounding of Spatial References with Relation Networkscited by 8
- 2023"Are you telling me to put glasses on the dog?'' Content-Grounded Annotation of Instruction Clarification Requests in the CoDraw Datasetcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023See and Think: Embodied Agent in Virtual Environment
- 2025BAP v2: An Enhanced Task Framework for Instruction Following in Minecraft Dialogues
- 2026MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft
- 2024TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft
- 2024STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft
- 2024MineLand: Simulating Large-Scale Multi-Agent Interactions with Limited Multimodal Senses and Physical Needs
- 2024Retrieval-Augmented Code Generation for Situated Action Generation: A Case Study on Minecraft
- 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraft