# Synthesizing Event-centric Knowledge Graphs of Daily Activities Using Virtual Space

arXiv: https://arxiv.org/abs/2307.16206 · HTML: https://mlsanity.com/paper/2307.16206/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 53 references that are in the map.
- First posted on arXiv 30 Jul 2023.

## Builds on (most cited first, 16 of 16)

- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Knowledge Graphs](https://mlsanity.com/paper/2003.02320/) (2020)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Domain-specific Knowledge Graphs: A survey](https://mlsanity.com/paper/2011.00235/) (2020)
- [MovieGraphs: Towards Understanding Human-Centric Situations from Videos](https://mlsanity.com/paper/1712.06761/) (2017)
- [ElderSim: A Synthetic Data Generation Platform for Human Action Recognition in Eldercare Applications](https://mlsanity.com/paper/2010.14742/) (2020)
- [Knowledge-based Embodied Question Answering](https://mlsanity.com/paper/2109.07872/) (2021)
- [EventKG - the Hub of Event Knowledge on the Web - and Biographical Timeline Generation](https://mlsanity.com/paper/1905.08794/) (2019)
- [Detecting Events of Daily Living Using Multimodal Data](https://mlsanity.com/paper/1905.09402/) (2019)

## Built on by (most cited first, 5 of 5)

- [Synthetic Multimodal Dataset for Empowering Safety and Well-being in Home Environments](https://mlsanity.com/paper/2401.14743/) (2024)
- [Multimodal Datasets and Benchmarks for Reasoning about Dynamic Spatio-Temporality in Everyday Environments](https://mlsanity.com/paper/2408.11347/) (2024)
- [VHAKG: A Multi-modal Knowledge Graph Based on Synchronized Multi-view Videos of Daily Activities](https://mlsanity.com/paper/2408.14895/) (2024)
- [A Video-grounded Dialogue Dataset and Metric for Event-driven Activities](https://mlsanity.com/paper/2501.18324/) (2025)
- [HOME-KGQA: A Benchmark Dataset for Multimodal Knowledge Graph Question Answering on Household Daily Activities](https://mlsanity.com/paper/2605.09348/) (2026)

## Abstract (arXiv)

Artificial intelligence (AI) is expected to be embodied in software agents, robots, and cyber-physical systems that can understand the various contextual information of daily life in the home environment to support human behavior and decision making in various situations. Scene graph and knowledge graph (KG) construction technologies have attracted much attention for knowledge-based embodied question answering meeting this expectation. However, collecting and managing real data on daily activities under various experimental conditions in a physical space are quite costly, and developing AI that understands the intentions and contexts is difficult. In the future, data from both virtual spaces, where conditions can be easily modified, and physical spaces, where conditions are difficult to change, are expected to be combined to analyze daily living activities. However, studies on the KG construction of daily activities using virtual space and their application have yet to progress. The potential and challenges must still be clarified to facilitate AI development for human daily life. Thus, this study proposes the VirtualHome2KG framework to generate synthetic KGs of daily life activities in virtual space. This framework augments both the synthetic video data of daily activities and the contextual semantic data corresponding to the video contents based on the proposed event-centric schema and virtual space simulation results. Therefore, context-aware data can be analyzed, and various applications that have conventionally been difficult to develop due to the insufficient availability of relevant data and semantic information can be developed. We also demonstrate herein the utility and potential of the proposed VirtualHome2KG framework through several use cases, including the analysis of daily activities by querying, embedding, and clustering, and fall risk detection among ...