# Multi-object event graph representation learning for Video Question Answering

arXiv: https://arxiv.org/abs/2409.07747 · HTML: https://mlsanity.com/paper/2409.07747/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 12 Sep 2024.

## Builds on (most cited first, 16 of 16)

- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Hierarchical Graph Representation Learning with Differentiable Pooling](https://mlsanity.com/paper/1806.08804/) (2018)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [Just Ask: Learning to Answer Questions from Millions of Narrated Videos](https://mlsanity.com/paper/2012.00451/) (2020)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)
- [Structural Deep Clustering Network](https://mlsanity.com/paper/2002.01633/) (2020)
- [Location-aware Graph Convolutional Networks for Video Question Answering](https://mlsanity.com/paper/2008.09105/) (2020)
- [Video as Conditional Graph Hierarchy for Multi-Granular Question Answering](https://mlsanity.com/paper/2112.06197/) (2021)
- [Video Graph Transformer for Video Question Answering](https://mlsanity.com/paper/2207.05342/) (2022)
- [MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answering](https://mlsanity.com/paper/2212.09522/) (2022)
- [Graph Pooling for Graph Neural Networks: Progress, Challenges, and Opportunities](https://mlsanity.com/paper/2204.07321/) (2022)

## Built on by (most cited first, 2 of 2)

- [HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation](https://mlsanity.com/paper/2411.18042/) (2024)
- [RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graph](https://mlsanity.com/paper/2505.03173/) (2025)

## Abstract (arXiv)

Video question answering (VideoQA) is a task to predict the correct answer to questions posed about a given video. The system must comprehend spatial and temporal relationships among objects extracted from videos to perform causal and temporal reasoning. While prior works have focused on modeling individual object movements using transformer-based methods, they falter when capturing complex scenarios involving multiple objects (e.g., "a boy is throwing a ball in a hoop"). We propose a contrastive language event graph representation learning method called CLanG to address this limitation. Aiming to capture event representations associated with multiple objects, our method employs a multi-layer GNN-cluster module for adversarial graph representation learning, enabling contrastive learning between the question text and its relevant multi-object event graph. Our method outperforms a strong baseline, achieving up to 2.2% higher accuracy on two challenging VideoQA datasets, NExT-QA and TGIF-QA-R. In particular, it is 2.8% better than baselines in handling causal and temporal questions, highlighting its strength in reasoning multiple object-based events.