# Alexa, play with robot: Introducing the First Alexa Prize SimBot Challenge on Embodied AI

arXiv: https://arxiv.org/abs/2308.05221 · HTML: https://mlsanity.com/paper/2308.05221/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 12 references that are in the map.
- First posted on arXiv 9 Aug 2023.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [TEACh: Task-driven Embodied Agents that Chat](https://mlsanity.com/paper/2110.00534/) (2021)
- [Episodic Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2105.06453/) (2021)
- [Alexa Arena: A User-Centric Interactive Platform for Embodied AI](https://mlsanity.com/paper/2303.01586/) (2023)
- [Alexa, Let's Work Together: Introducing the First Alexa Prize TaskBot Challenge on Conversational Task Assistance](https://mlsanity.com/paper/2209.06321/) (2022)

## Built on by (most cited first, 5 of 5)

- [Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects](https://mlsanity.com/paper/2401.03428/) (2024)
- [Instance Tracking in 3D Scenes from Egocentric Videos](https://mlsanity.com/paper/2312.04117/) (2023)
- [Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks](https://mlsanity.com/paper/2407.03967/) (2024)
- [The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcare](https://mlsanity.com/paper/2411.03287/) (2024)
- [Amazon Nova AI Challenge -- Trusted AI: Advancing secure, AI-assisted software development](https://mlsanity.com/paper/2508.10108/) (2025)

## Abstract (arXiv)

The Alexa Prize program has empowered numerous university students to explore, experiment, and showcase their talents in building conversational agents through challenges like the SocialBot Grand Challenge and the TaskBot Challenge. As conversational agents increasingly appear in multimodal and embodied contexts, it is important to explore the affordances of conversational interaction augmented with computer vision and physical embodiment. This paper describes the SimBot Challenge, a new challenge in which university teams compete to build robot assistants that complete tasks in a simulated physical environment. This paper provides an overview of the SimBot Challenge, which included both online and offline challenge phases. We describe the infrastructure and support provided to the teams including Alexa Arena, the simulated environment, and the ML toolkit provided to teams to accelerate their building of vision and language models. We summarize the approaches the participating teams took to overcome research challenges and extract key lessons learned. Finally, we provide analysis of the performance of the competing SimBots during the competition.