# Multimodal Datasets and Benchmarks for Reasoning about Dynamic Spatio-Temporality in Everyday Environments

arXiv: https://arxiv.org/abs/2408.11347 · HTML: https://mlsanity.com/paper/2408.11347/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 21 Aug 2024.

## Builds on (most cited first, 12 of 12)

- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution](https://mlsanity.com/paper/2107.05612/) (2021)
- [Retrospectives on the Embodied AI Workshop](https://mlsanity.com/paper/2210.06849/) (2022)
- [A call for embodied AI](https://mlsanity.com/paper/2402.03824/) (2024)
- [Synthesizing Event-centric Knowledge Graphs of Daily Activities Using Virtual Space](https://mlsanity.com/paper/2307.16206/) (2023)
- [Synthetic Multimodal Dataset for Empowering Safety and Well-being in Home Environments](https://mlsanity.com/paper/2401.14743/) (2024)

## Built on by (most cited first, 1 of 1)

- [VUDG: A Dataset for Video Understanding Domain Generalization](https://mlsanity.com/paper/2505.24346/) (2025)

## Abstract (arXiv)

We used a 3D simulator to create artificial video data with standardized annotations, aiming to aid in the development of Embodied AI. Our question answering (QA) dataset measures the extent to which a robot can understand human behavior and the environment in a home setting. Preliminary experiments suggest our dataset is useful in measuring AI's comprehension of daily life. endabstract