# Spoken Language Interaction with Robots: Research Issues and Recommendations, Report from the NSF Future Directions Workshop

arXiv: https://arxiv.org/abs/2011.05533 · HTML: https://mlsanity.com/paper/2011.05533/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2020.

## Built on by (most cited first, 15 of 15)

- [Core Challenges in Embodied Vision-Language Planning](https://mlsanity.com/paper/2106.13948/) (2021)
- [Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems](https://mlsanity.com/paper/2205.15060/) (2022)
- [DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiences](https://mlsanity.com/paper/2406.03008/) (2024)
- [Morae: Proactively Pausing UI Agents for User Choices](https://mlsanity.com/paper/2508.21456/) (2025)
- [Natural Language Processing for Smart Healthcare](https://mlsanity.com/paper/2110.15803/) (2021)
- [Multimodal Speech Recognition for Language-Guided Embodied Agents](https://mlsanity.com/paper/2302.14030/) (2023)
- [A Collection of Pragmatic-Similarity Judgments over Spoken Dialog Utterances](https://mlsanity.com/paper/2403.14808/) (2024)
- [Which Prosodic Features Matter Most for Pragmatics?](https://mlsanity.com/paper/2408.13240/) (2024)
- [The State of SLIVAR: What's next for robots, human-robot interaction, and (spoken) dialogue systems?](https://mlsanity.com/paper/2108.10931/) (2021)
- [Bridging the Gap: Using Deep Acoustic Representations to Learn Grounded Language from Percepts and Raw Speech](https://mlsanity.com/paper/2112.13758/) (2021)
- [Whither the Priors for (Vocal) Interactivity?](https://mlsanity.com/paper/2203.08578/) (2022)
- [Core Challenges in Embodied Vision-Language Planning](https://mlsanity.com/paper/2304.02738/) (2023)
- [Emphasis Sensitivity in Speech Representations](https://mlsanity.com/paper/2508.11566/) (2025)
- [Toward Self-Repairing Ubiquitous Robots Using Goal-Oriented Agentic AI in Human-Robot Interactions](https://mlsanity.com/paper/2609.26155/) (2026)
- [Toward User-Mediated Self-Repair in Ubiquitous Robots Through Goal-Oriented Agentic AI](https://mlsanity.com/paper/2609.26157/) (2026)

## Abstract (arXiv)

With robotics rapidly advancing, more effective human-robot interaction is increasingly needed to realize the full potential of robots for society. While spoken language must be part of the solution, our ability to provide spoken language interaction capabilities is still very limited. The National Science Foundation accordingly convened a workshop, bringing together speech, language, and robotics researchers to discuss what needs to be done. The result is this report, in which we identify key scientific and engineering advances needed. Our recommendations broadly relate to eight general themes. First, meeting human needs requires addressing new challenges in speech technology and user experience design. Second, this requires better models of the social and interactive aspects of language use. Third, for robustness, robots need higher-bandwidth communication with users and better handling of uncertainty, including simultaneous consideration of multiple hypotheses and goals. Fourth, more powerful adaptation methods are needed, to enable robots to communicate in new environments, for new tasks, and with diverse user populations, without extensive re-engineering or the collection of massive training data. Fifth, since robots are embodied, speech should function together with other communication modalities, such as gaze, gesture, posture, and motion. Sixth, since robots operate in complex environments, speech components need access to rich yet efficient representations of what the robot knows about objects, locations, noise sources, the user, and other humans. Seventh, since robots operate in real time, their speech and language processing components must also. Eighth, in addition to more research, we need more work on infrastructure and resources, including shareable software modules and internal interfaces, inexpensive hardware, baseline systems, and diverse corpora.