# From Pixels to Graphs: using Scene and Knowledge Graphs for HD-EPIC VQA Challenge

arXiv: https://arxiv.org/abs/2506.08553 · HTML: https://mlsanity.com/paper/2506.08553/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 25 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Builds on (most cited first, 17 of 17)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera](https://mlsanity.com/paper/1910.02527/) (2019)
- [The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines](https://mlsanity.com/paper/2005.00343/) (2020)
- [Jasper and Stella: distillation of SOTA embedding models](https://mlsanity.com/paper/2412.19048/) (2024)
- [Action Scene Graphs for Long-Form Understanding of Egocentric Videos](https://mlsanity.com/paper/2312.03391/) (2023)
- [Neuro-Symbolic Visual Reasoning: Disentangling "Visual" from "Reasoning"](https://mlsanity.com/paper/2006.11524/) (2020)
- [Learning by Asking Questions](https://mlsanity.com/paper/1712.01238/) (2017)
- [Learning Situation Hyper-Graphs for Video Question Answering](https://mlsanity.com/paper/2304.08682/) (2023)
- [Augmented Commonsense Knowledge for Remote Object Grounding](https://mlsanity.com/paper/2406.01256/) (2024)
- [Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation](https://mlsanity.com/paper/2411.13059/) (2024)
- [ALGO: Object-Grounded Visual Commonsense Reasoning for Open-World Egocentric Action Recognition](https://mlsanity.com/paper/2406.05722/) (2024)

## Built on by (most cited first, 6 of 6)

- [Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges](https://mlsanity.com/paper/2511.13261/) (2025)
- [Optimizing Multimodal LLMs for Egocentric Video Understanding: A Solution for the HD-EPIC VQA Challenge](https://mlsanity.com/paper/2601.10228/) (2026)
- [Leveraging Gaze and Set-of-Mark in VLLMs for Human-Object Interaction Anticipation from Egocentric Videos](https://mlsanity.com/paper/2604.03667/) (2026)
- [From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models](https://mlsanity.com/paper/2606.15417/) (2026)
- [Learning to Deny: Action Denial in Multimodal Large Language Models](https://mlsanity.com/paper/2606.31187/) (2026)
- [Never Look Back: Understanding Persistence in 3D Object Memory from Egocentric Videos](https://mlsanity.com/paper/2610.10538/) (2026)

## Abstract (arXiv)

This report presents SceneNet and KnowledgeNet, our approaches developed for the HD-EPIC VQA Challenge 2025. SceneNet leverages scene graphs generated with a multi-modal large language model (MLLM) to capture fine-grained object interactions, spatial relationships, and temporally grounded events. In parallel, KnowledgeNet incorporates ConceptNet's external commonsense knowledge to introduce high-level semantic connections between entities, enabling reasoning beyond directly observable visual evidence. Each method demonstrates distinct strengths across the seven categories of the HD-EPIC benchmark, and their combination within our framework results in an overall accuracy of 44.21% on the challenge, highlighting its effectiveness for complex egocentric VQA tasks.