# RescueBench: Can Embodied Agents Save Lives in the Wild ?

arXiv: https://arxiv.org/abs/2606.01848 · HTML: https://mlsanity.com/paper/2606.01848/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Builds on (most cited first, 26 of 26)

- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2004.02857/) (2020)
- [Speaker-Follower Models for Vision-and-Language Navigation](https://mlsanity.com/paper/1806.02724/) (2018)
- [NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration](https://mlsanity.com/paper/2310.07896/) (2023)
- [Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots](https://mlsanity.com/paper/2310.13724/) (2023)
- [ViNT: A Foundation Model for Visual Navigation](https://mlsanity.com/paper/2306.14846/) (2023)
- [EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents](https://mlsanity.com/paper/2502.09560/) (2025)
- [SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation](https://mlsanity.com/paper/2410.08189/) (2024)
- [PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks](https://mlsanity.com/paper/2411.00081/) (2024)
- [TrackVLA: Embodied Visual Tracking in the Wild](https://mlsanity.com/paper/2505.23189/) (2025)
- [COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models](https://mlsanity.com/paper/2409.15146/) (2024)
- [CityWalker: Learning Embodied Urban Navigation from Web-Scale Videos](https://mlsanity.com/paper/2411.17820/) (2024)
- [OmniNav: A Unified Framework for Prospective Exploration and Visual-Language Navigation](https://mlsanity.com/paper/2509.25687/) (2025)
- [HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environments](https://mlsanity.com/paper/2401.12975/) (2024)
- [UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI](https://mlsanity.com/paper/2412.20977/) (2024)
- [Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RL](https://mlsanity.com/paper/2404.09857/) (2024)
- [TrackVLA++: Unleashing Reasoning and Memory Capabilities in VLA Models for Embodied Visual Tracking](https://mlsanity.com/paper/2510.07134/) (2025)
- [Proactive Multi-Camera Collaboration For 3D Human Pose Estimation](https://mlsanity.com/paper/2303.03767/) (2023)
- [Learning Semantic-Agnostic and Spatial-Aware Representation for Generalizable Visual-Audio Navigation](https://mlsanity.com/paper/2304.10773/) (2023)
- [ROCKET-2: Steering Visuomotor Policy via Cross-View Goal Alignment](https://mlsanity.com/paper/2503.02505/) (2025)
- [VLM Can Be a Good Assistant: Enhancing Embodied Visual Tracking with Self-Improving Vision-Language Models](https://mlsanity.com/paper/2505.20718/) (2025)
- [Communication-Efficient Desire Alignment for Embodied Agent-Human Adaptation](https://mlsanity.com/paper/2505.22503/) (2025)
- [AdaTracker: Learning Adaptive In-Context Policy for Cross-Embodiment Active Visual Tracking](https://mlsanity.com/paper/2604.20305/) (2026)

## Abstract (arXiv)

Search-and-rescue (SAR) requires embodied agents to explore unfamiliar environments under multimodal uncertainty, perform multi-stage interactions, and retrieve spatial memory over long horizons. Existing benchmarks typically evaluate these capabilities in isolation, leaving unclear how failures compound when they must be composed in realistic workflows. We introduce RescueBench, a photo-realistic diagnostic benchmark that instantiates SAR as a four-stage pipeline: multimodal exploration, target rescue, memory-guided return, and final handoff. By combining sequential task composition with stage-level evaluation, RescueBench enables analysis of how exploration and memory failures propagate through embodied rescue workflows. It contains five progressive difficulty levels that vary in environmental complexity, clue ambiguity, and spatial hierarchy, along with an automatic episode generation and annotation pipeline for scalable evaluation and training. We evaluate seven baselines, an oracle reference, and human players, showing that no baselines complete the full task at the greatest difficulty. Stage-level diagnosis identifies autonomous exploration as the dominant failure mode and spatial memory as a second, independent bottleneck, suggesting that these limitations are not resolved by current topological visual-language navigation or map-based methods. Code is available in https://github.com/wukui-muc/RescueBench