# R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video

arXiv: https://arxiv.org/abs/2608.11017 · HTML: https://mlsanity.com/paper/2608.11017/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 42 references that are in the map.
- First posted on arXiv 11 Aug 2026.

## Builds on (most cited first, 27 of 27)

- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [SAM 3D: 3Dfy Anything in Images](https://mlsanity.com/paper/2511.16624/) (2025)
- [Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions](https://mlsanity.com/paper/2004.03967/) (2020)
- [Project Aria: A New Tool for Egocentric Multi-Modal AI Research](https://mlsanity.com/paper/2308.13561/) (2023)
- [SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences](https://mlsanity.com/paper/2103.14898/) (2021)
- [Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships](https://mlsanity.com/paper/2402.12259/) (2024)
- [Grounded Question-Answering in Long Egocentric Videos](https://mlsanity.com/paper/2312.06505/) (2023)
- [Knowledge Graph-Guided Retrieval Augmented Generation](https://mlsanity.com/paper/2502.06864/) (2025)
- [Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces](https://mlsanity.com/paper/2503.19199/) (2025)
- [Grounded Multi-Hop VideoQA in Long-Form Egocentric Videos](https://mlsanity.com/paper/2408.14469/) (2024)
- [VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloud](https://mlsanity.com/paper/2303.14408/) (2023)
- [Incremental 3D Semantic Scene Graph Prediction from RGB Sequences](https://mlsanity.com/paper/2305.02743/) (2023)
- [GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality](https://mlsanity.com/paper/2404.08213/) (2024)
- [AMEGO: Active Memory from long EGOcentric videos](https://mlsanity.com/paper/2409.10917/) (2024)
- [Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labeling](https://mlsanity.com/paper/2308.05081/) (2023)
- [ProMemAssist: Exploring Timely Proactive Assistance Through Working Memory Modeling in Multi-Modal Wearable Devices](https://mlsanity.com/paper/2507.21378/) (2025)
- [AiGet: Transforming Everyday Moments into Hidden Knowledge Discovery with AI Assistance on Smart Glasses](https://mlsanity.com/paper/2501.16240/) (2025)
- [Towards Robust Monocular Depth Estimation in Non-Lambertian Surfaces](https://mlsanity.com/paper/2408.06083/) (2024)
- [Phys-Liquid: A Physics-Informed Dataset for Estimating 3D Geometry and Volume of Transparent Deformable Liquids](https://mlsanity.com/paper/2511.11077/) (2025)
- [Find Beauty in the Rare: Contrastive Composition Feature Clustering for Nontrivial Cropping Box Regression](https://mlsanity.com/paper/2302.08662/) (2023)
- [Imaging for All-Day Wearable Smart Glasses](https://mlsanity.com/paper/2504.13060/) (2025)
- [SAMJAM: Zero-Shot Video Scene Graph Generation for Egocentric Kitchen Videos](https://mlsanity.com/paper/2504.07867/) (2025)
- [ResilPhase: Plug-and-Play Phase Mapping and Noise-Resilient Macro-Trajectory Extrapolation for Diffusion Acceleration](https://mlsanity.com/paper/2606.26769/) (2026)
- [TransBiolab: A Real-World Multi-View Dataset of Cluttered Transparent Biomedical Objects](https://mlsanity.com/paper/2607.21071/) (2026)

## Built on by (most cited first, 1 of 1)

- [Never Look Back: Understanding Persistence in 3D Object Memory from Egocentric Videos](https://mlsanity.com/paper/2610.10538/) (2026)

## Abstract (arXiv)

Long-horizon egocentric video is a rich substrate for wearable AI assistants, but object-centric questions such as where an item was moved, when it last changed state, or why it was relocated remain difficult because caption- and transcript-based memories rarely preserve persistent object identity or structured spatial change. Existing long-video QA methods mainly emphasize temporal grounding and clip retrieval, while prior 3D scene-graph methods typically assume stronger geometry than free-motion wearable RGB video provides, including point clouds, RGB-D input, posed views, sparse reconstruction, or reconstructed scenes. R4DSG introduces a relative 4D scene graph memory for long egocentric video. Instead of storing raw graph sequences, R4DSG converts video into compact queryable memory entries indexed by time, place, persistent objects, anchor-relative change, and local interaction context. The main idea is to separate stable anchors from dynamic objects, maintain persistent object identity across frames, and represent object state through anchor-relative transitions rather than a globally aligned world model. Built on recent RGB-only advances in promptable video segmentation, temporal propagation, and relative 3D lifting, the method produces a retrieval-ready memory directly usable for long-horizon question answering. Evaluation on a 255-question object-related subset from EgoLifeQA shows, under question-only retrieval, a 6.7-point overall gain over EgoRAG-Text and a 12.5-point gain on when questions, which highlights the value of temporally organized object memory. These results position relative 4D scene graphs as a practical memory substrate for wearable assistants, AR systems, and embodied multimedia agents. GitHub Page: https://dualtransparency.github.io/R4DSG/.