# Multimodal Speech Recognition for Language-Guided Embodied Agents

arXiv: https://arxiv.org/abs/2302.14030 · HTML: https://mlsanity.com/paper/2302.14030/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 27 Feb 2023.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding](https://mlsanity.com/paper/2010.07954/) (2020)
- [REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments](https://mlsanity.com/paper/1904.10151/) (2019)
- [Vision-and-Dialog Navigation](https://mlsanity.com/paper/1907.04957/) (2019)
- [Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments](https://mlsanity.com/paper/1811.12354/) (2018)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [Episodic Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2105.06453/) (2021)
- [How2: A Large-scale Dataset for Multimodal Language Understanding](https://mlsanity.com/paper/1811.00347/) (2018)
- [FILM: Following Instructions in Language with Modular Methods](https://mlsanity.com/paper/2110.07342/) (2021)
- [Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions](https://mlsanity.com/paper/2203.12667/) (2022)
- [A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution](https://mlsanity.com/paper/2107.05612/) (2021)
- [Executing Instructions in Situated Collaborative Interactions](https://mlsanity.com/paper/1910.03655/) (2019)
- [Probing the Need for Visual Context in Multimodal Machine Translation](https://mlsanity.com/paper/1903.08678/) (2019)
- [The RobotSlang Benchmark: Dialog-guided Robot Localization and Navigation](https://mlsanity.com/paper/2010.12639/) (2020)
- [AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments](https://mlsanity.com/paper/2210.07940/) (2022)
- [Spoken Language Interaction with Robots: Research Issues and Recommendations, Report from the NSF Future Directions Workshop](https://mlsanity.com/paper/2011.05533/) (2020)
- [End-to-End Multimodal Speech Recognition](https://mlsanity.com/paper/1804.09713/) (2018)
- [Multimodal Grounding for Sequence-to-Sequence Speech Recognition](https://mlsanity.com/paper/1811.03865/) (2018)
- [Multimodal Speech Recognition with Unstructured Audio Masking](https://mlsanity.com/paper/2010.08642/) (2020)
- [Looking Enhances Listening: Recovering Missing Speech Using Images](https://mlsanity.com/paper/2002.05639/) (2020)
- [Fine-Grained Grounding for Multimodal Speech Recognition](https://mlsanity.com/paper/2010.02384/) (2020)
- [Analyzing Utility of Visual Context in Multimodal Speech Recognition Under Noisy Conditions](https://mlsanity.com/paper/1907.00477/) (2019)

## Built on by (most cited first, 3 of 3)

- [VHASR: A Multimodal Speech Recognition System With Vision Hotwords](https://mlsanity.com/paper/2410.00822/) (2024)
- [Exploring the Potential of Multimodal LLM with Knowledge-Intensive Multimodal ASR](https://mlsanity.com/paper/2406.10880/) (2024)
- [Layover or Direct Flight: Rethinking Audio-Guided Image Segmentation](https://mlsanity.com/paper/2511.22025/) (2025)

## Abstract (arXiv)

Benchmarks for language-guided embodied agents typically assume text-based instructions, but deployed agents will encounter spoken instructions. While Automatic Speech Recognition (ASR) models can bridge the input gap, erroneous ASR transcripts can hurt the agents' ability to complete tasks. In this work, we propose training a multimodal ASR model to reduce errors in transcribing spoken instructions by considering the accompanying visual context. We train our model on a dataset of spoken instructions, synthesized from the ALFRED task completion dataset, where we simulate acoustic noise by systematically masking spoken words. We find that utilizing visual observations facilitates masked word recovery, with multimodal ASR models recovering up to 30% more masked words than unimodal baselines. We also find that a text-trained embodied agent successfully completes tasks more often by following transcribed instructions from multimodal ASR models. github.com/Cylumn/embodied-multimodal-asr