# SAMJAM: Zero-Shot Video Scene Graph Generation for Egocentric Kitchen Videos

arXiv: https://arxiv.org/abs/2504.07867 · HTML: https://mlsanity.com/paper/2504.07867/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 10 Apr 2025.

## Builds on (most cited first, 17 of 17)

- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Panoptic Video Scene Graph Generation](https://mlsanity.com/paper/2311.17058/) (2023)
- [Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis](https://mlsanity.com/paper/2409.00106/) (2024)
- [CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videos](https://mlsanity.com/paper/2406.01029/) (2024)
- [Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generation](https://mlsanity.com/paper/2309.13237/) (2023)
- [Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation](https://mlsanity.com/paper/2412.07160/) (2024)
- [Weakly Supervised Video Scene Graph Generation via Natural Language Supervision](https://mlsanity.com/paper/2502.15370/) (2025)
- [Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos](https://mlsanity.com/paper/2306.15644/) (2023)

## Built on by (most cited first, 4 of 4)

- [HiM2SAM: Enhancing SAM2 with Hierarchical Motion Estimation and Memory Optimization towards Long-term Tracking](https://mlsanity.com/paper/2507.07603/) (2025)
- [Situat3DChange: Situated 3D Change Understanding Dataset for Multimodal Large Language Model](https://mlsanity.com/paper/2510.11509/) (2025)
- [Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answering](https://mlsanity.com/paper/2510.01483/) (2025)
- [R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video](https://mlsanity.com/paper/2608.11017/) (2026)

## Abstract (arXiv)

Video Scene Graph Generation (VidSGG) is an important topic in understanding dynamic kitchen environments. Current models for VidSGG require extensive training to produce scene graphs. Recently, Vision Language Models (VLM) and Vision Foundation Models (VFM) have demonstrated impressive zero-shot capabilities in a variety of tasks. However, VLMs like Gemini struggle with the dynamics for VidSGG, failing to maintain stable object identities across frames. To overcome this limitation, we propose SAMJAM, a zero-shot pipeline that combines SAM2's temporal tracking with Gemini's semantic understanding. SAM2 also improves upon Gemini's object grounding by producing more accurate bounding boxes. In our method, we first prompt Gemini to generate a frame-level scene graph. Then, we employ a matching algorithm to map each object in the scene graph with a SAM2-generated or SAM2-propagated mask, producing a temporally-consistent scene graph in dynamic environments. Finally, we repeat this process again in each of the following frames. We empirically demonstrate that SAMJAM outperforms Gemini by 8.33% in mean recall on the EPIC-KITCHENS and EPIC-KITCHENS-100 datasets.