# Retrospectives on the Embodied AI Workshop

arXiv: https://arxiv.org/abs/2210.06849 · HTML: https://mlsanity.com/paper/2210.06849/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 169 of its 220 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 30 of 169)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Perceiver IO: A General Architecture for Structured Inputs & Outputs](https://mlsanity.com/paper/2107.14795/) (2021)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)

## Built on by (most cited first, 30 of 46)

- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Building Cooperative Embodied Agents Modularly with Large Language Models](https://mlsanity.com/paper/2307.02485/) (2023)
- [Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?](https://mlsanity.com/paper/2303.18240/) (2023)
- [ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2304.03047/) (2023)
- [ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes](https://mlsanity.com/paper/2304.04321/) (2023)
- [Principles and Guidelines for Evaluating Social Robot Navigation Algorithms](https://mlsanity.com/paper/2306.16740/) (2023)
- [Instance-Specific Image Goal Navigation: Training Embodied Agents to Find Object Instances](https://mlsanity.com/paper/2211.15876/) (2022)
- [Navigating to Objects Specified by Images](https://mlsanity.com/paper/2304.01192/) (2023)
- [Learning Human-to-Robot Handovers from Point Clouds](https://mlsanity.com/paper/2303.17592/) (2023)
- [OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics](https://mlsanity.com/paper/2401.12202/) (2024)
- [Vid2Sim: Realistic and Interactive Simulation from Video for Urban Navigation](https://mlsanity.com/paper/2501.06693/) (2025)
- [Pretrained Language Models as Visual Planners for Human Assistance](https://mlsanity.com/paper/2304.09179/) (2023)
- [Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approach](https://mlsanity.com/paper/2306.03604/) (2023)
- [Last-Mile Embodied Visual Navigation](https://mlsanity.com/paper/2211.11746/) (2022)
- [PEANUT: Predicting and Navigating to Unseen Targets](https://mlsanity.com/paper/2212.02497/) (2022)
- [Causal Triplet: An Open Challenge for Intervention-centric Causal Representation Learning](https://mlsanity.com/paper/2301.05169/) (2023)
- [MetaUrban: An Embodied AI Simulation Platform for Urban Micromobility](https://mlsanity.com/paper/2407.08725/) (2024)
- [Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI](https://mlsanity.com/paper/2311.03783/) (2023)
- [Enter the Mind Palace: Reasoning and Planning for Long-term Active Embodied Question Answering](https://mlsanity.com/paper/2507.12846/) (2025)
- [Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge](https://mlsanity.com/paper/2411.01796/) (2024)
- [MOPA: Modular Object Navigation with PointGoal Agents](https://mlsanity.com/paper/2304.03696/) (2023)
- [AutoNeRF: Training Implicit Scene Representations with Autonomous Agents](https://mlsanity.com/paper/2304.11241/) (2023)
- [HabiCrowd: A High Performance Simulator for Crowd-Aware Visual Navigation](https://mlsanity.com/paper/2306.11377/) (2023)
- [Exploitation-Guided Exploration for Semantic Embodied Navigation](https://mlsanity.com/paper/2311.03357/) (2023)
- [Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge](https://mlsanity.com/paper/2407.06939/) (2024)
- [Task-oriented Sequential Grounding and Navigation in 3D Scenes](https://mlsanity.com/paper/2408.04034/) (2024)
- [Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Prediction](https://mlsanity.com/paper/2405.02821/) (2024)
- [Exploiting Proximity-Aware Tasks for Embodied Social Navigation](https://mlsanity.com/paper/2212.00767/) (2022)
- [Learning-Augmented Model-Based Planning for Visual Exploration](https://mlsanity.com/paper/2211.07898/) (2022)
- [Following the Human Thread in Social Navigation](https://mlsanity.com/paper/2404.11327/) (2024)

## Abstract (arXiv)

We present a retrospective on the state of Embodied AI research. Our analysis focuses on 13 challenges presented at the Embodied AI Workshop at CVPR. These challenges are grouped into three themes: (1) visual navigation, (2) rearrangement, and (3) embodied vision-and-language. We discuss the dominant datasets within each theme, evaluation metrics for the challenges, and the performance of state-of-the-art models. We highlight commonalities between top approaches to the challenges and identify potential future directions for Embodied AI research.