# Video Relation Detection with Trajectory-aware Multi-modal Features

arXiv: https://arxiv.org/abs/2101.08165 · HTML: https://mlsanity.com/paper/2101.08165/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 19 references that are in the map.
- First posted on arXiv 20 Jan 2021.

## Builds on (most cited first, 10 of 10)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [ResNeSt: Split-Attention Networks](https://mlsanity.com/paper/2004.08955/) (2020)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection](https://mlsanity.com/paper/1912.12898/) (2019)
- [Seq-NMS for Video Object Detection](https://mlsanity.com/paper/1602.08465/) (2016)
- [Video Relationship Reasoning using Gated Spatio-Temporal Energy Graph](https://mlsanity.com/paper/1903.10547/) (2019)

## Built on by (most cited first, 8 of 8)

- [Classification-Then-Grounding: Reformulating Video Scene Graphs as Temporal Bipartite Graphs](https://mlsanity.com/paper/2112.04222/) (2021)
- [Social Fabric: Tubelet Compositions for Video Relation Detection](https://mlsanity.com/paper/2108.08363/) (2021)
- [Video Relation Detection via Tracklet based Visual Transformer](https://mlsanity.com/paper/2108.08669/) (2021)
- [Diagnosing Errors in Video Relation Detectors](https://mlsanity.com/paper/2110.13110/) (2021)
- [Decoupled Spatial Temporal Graphs for Generic Visual Grounding](https://mlsanity.com/paper/2103.10191/) (2021)
- [Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation](https://mlsanity.com/paper/2306.10122/) (2023)
- [What and When to Look?: Temporal Span Proposal Network for Video Relation Detection](https://mlsanity.com/paper/2107.07154/) (2021)
- [Rethinking Multi-Modal Alignment in Video Question Answering from Feature and Sample Perspectives](https://mlsanity.com/paper/2204.11544/) (2022)

## Abstract (arXiv)

Video relation detection problem refers to the detection of the relationship between different objects in videos, such as spatial relationship and action relationship. In this paper, we present video relation detection with trajectory-aware multi-modal features to solve this task. Considering the complexity of doing visual relation detection in videos, we decompose this task into three sub-tasks: object detection, trajectory proposal and relation prediction. We use the state-of-the-art object detection method to ensure the accuracy of object trajectory detection and multi-modal feature representation to help the prediction of relation between objects. Our method won the first place on the video relation detection task of Video Relation Understanding Grand Challenge in ACM Multimedia 2020 with 11.74% mAP, which surpasses other methods by a large margin.