# HomeEmergency -- Using Audio to Find and Respond to Emergencies in the Home

arXiv: https://arxiv.org/abs/2504.01089 · HTML: https://mlsanity.com/paper/2504.01089/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Vision-and-Dialog Navigation](https://mlsanity.com/paper/1907.04957/) (2019)
- [ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation](https://mlsanity.com/paper/2007.04954/) (2020)
- [3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans](https://mlsanity.com/paper/2002.06289/) (2020)
- [Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2101.06894/) (2021)
- [Look, Listen, and Act: Towards Audio-Visual Embodied Navigation](https://mlsanity.com/paper/1912.11684/) (2019)
- [Semantic Audio-Visual Navigation](https://mlsanity.com/paper/2012.11583/) (2020)
- [Clio: Real-time Task-Driven Open-Set 3D Scene Graphs](https://mlsanity.com/paper/2404.13696/) (2024)
- [Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation](https://mlsanity.com/paper/2303.03480/) (2023)
- [DialFRED: Dialogue-Enabled Agents for Embodied Instruction Following](https://mlsanity.com/paper/2202.13330/) (2022)
- [Retrospectives on the Embodied AI Workshop](https://mlsanity.com/paper/2210.06849/) (2022)
- [Improving Grounded Natural Language Understanding through Human-Robot Dialog](https://mlsanity.com/paper/1903.00122/) (2019)
- [Ground then Navigate: Language-guided Navigation in Dynamic Scenes](https://mlsanity.com/paper/2209.11972/) (2022)
- [Proactive Robot Assistance via Spatio-Temporal Object Modeling](https://mlsanity.com/paper/2211.15501/) (2022)
- [Long-Term Human Trajectory Prediction using 3D Dynamic Scene Graphs](https://mlsanity.com/paper/2405.00552/) (2024)
- [Finding Fallen Objects Via Asynchronous Audio-Visual Integration](https://mlsanity.com/paper/2207.03483/) (2022)
- [Safe Navigation with Human Instructions in Complex Scenes](https://mlsanity.com/paper/1809.04280/) (2018)
- ["Don't forget to put the milk back!" Dataset for Enabling Embodied Agents to Detect Anomalous Situations](https://mlsanity.com/paper/2404.08827/) (2024)
- [Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Prediction](https://mlsanity.com/paper/2405.02821/) (2024)
- [Can a Robot Trust You? A DRL-Based Approach to Trust-Driven Human-Guided Navigation](https://mlsanity.com/paper/2011.00554/) (2020)
- [Multi-goal Audio-visual Navigation using Sound Direction Map](https://mlsanity.com/paper/2308.00219/) (2023)
- [VLPG-Nav: Object Navigation Using Visual Language Pose Graph and Object Localization Probability Maps](https://mlsanity.com/paper/2408.08301/) (2024)

## Abstract (arXiv)

In the United States alone accidental home deaths exceed 128,000 per year. Our work aims to enable home robots who respond to emergency scenarios in the home, preventing injuries and deaths. We introduce a new dataset of household emergencies based in the ThreeDWorld simulator. Each scenario in our dataset begins with an instantaneous or periodic sound which may or may not be an emergency. The agent must navigate the multi-room home scene using prior observations, alongside audio signals and images from the simulator, to determine if there is an emergency or not. In addition to our new dataset, we present a modular approach for localizing and identifying potential home emergencies. Underpinning our approach is a novel probabilistic dynamic scene graph (P-DSG), where our key insight is that graph nodes corresponding to agents can be represented with a probabilistic edge. This edge, when refined using Bayesian inference, enables efficient and effective localization of agents in the scene. We also utilize multi-modal vision-language models (VLMs) as a component in our approach, determining object traits (e.g. flammability) and identifying emergencies. We present a demonstration of our method completing a real-world version of our task on a consumer robot, showing the transferability of both our task and our method. Our dataset will be released to the public upon this papers publication.