# Solving Dialogue Grounding Embodied Task in a Simulated Environment using Further Masked Language Modeling

arXiv: https://arxiv.org/abs/2306.12387 · HTML: https://mlsanity.com/paper/2306.12387/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 30 references that are in the map.
- First posted on arXiv 21 Jun 2023.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Vision-and-Dialog Navigation](https://mlsanity.com/paper/1907.04957/) (2019)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [GuessWhat?! Visual object discovery through multi-modal dialogue](https://mlsanity.com/paper/1611.08481/) (2016)
- [Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning](https://mlsanity.com/paper/1909.01871/) (2019)
- [Mapping Instructions and Visual Observations to Actions with Reinforcement Learning](https://mlsanity.com/paper/1704.08795/) (2017)
- [Executing Instructions in Situated Collaborative Interactions](https://mlsanity.com/paper/1910.03655/) (2019)
- [StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts](https://mlsanity.com/paper/2204.08292/) (2022)
- [Just Ask:An Interactive Learning Framework for Vision and Language Navigation](https://mlsanity.com/paper/1912.00915/) (2019)
- [RMM: A Recursive Mental Model for Dialog Navigation](https://mlsanity.com/paper/2005.00728/) (2020)
- [Learning to Execute Actions or Ask Clarification Questions](https://mlsanity.com/paper/2204.08373/) (2022)
- [Representation Learning for Grounded Spatial Reasoning](https://mlsanity.com/paper/1707.03938/) (2017)
- [Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner](https://mlsanity.com/paper/2305.01711/) (2023)
- [Rethinking Semi-supervised Learning with Language Models](https://mlsanity.com/paper/2305.13002/) (2023)
- [Situated Mapping of Sequential Instructions to Actions with Single-step Reward Observation](https://mlsanity.com/paper/1805.10209/) (2018)
- [Robust and Interpretable Grounding of Spatial References with Relation Networks](https://mlsanity.com/paper/2005.00696/) (2020)
- [When and What to Ask Through World States and Text Instructions: IGLU NLP Challenge Solution](https://mlsanity.com/paper/2305.05754/) (2023)
- ["Are you telling me to put glasses on the dog?'' Content-Grounded Annotation of Instruction Clarification Requests in the CoDraw Dataset](https://mlsanity.com/paper/2306.02377/) (2023)

## Abstract (arXiv)

Enhancing AI systems with efficient communication skills that align with human understanding is crucial for their effective assistance to human users. Proactive initiatives from the system side are needed to discern specific circumstances and interact aptly with users to solve these scenarios. In this research, we opt for a collective building assignment taken from the Minecraft dataset. Our proposed method employs language modeling to enhance task understanding through state-of-the-art (SOTA) methods using language models. These models focus on grounding multi-modal understandinging and task-oriented dialogue comprehension tasks. This focus aids in gaining insights into how well these models interpret and respond to a variety of inputs and tasks. Our experimental results provide compelling evidence of the superiority of our proposed method. This showcases a substantial improvement and points towards a promising direction for future research in this domain.