# Temporal Dynamic Graph LSTM for Action-driven Video Object Detection

arXiv: https://arxiv.org/abs/1708.00666 · HTML: https://mlsanity.com/paper/1708.00666/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2017.

## Built on by (most cited first, 23 of 23)

- [Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cut](https://mlsanity.com/paper/2202.11539/) (2022)
- [TokenCut: Segmenting Objects in Images and Videos with Self-supervised Transformer and Normalized Cut](https://mlsanity.com/paper/2209.00383/) (2022)
- [A Survey on Embedding Dynamic Graphs](https://mlsanity.com/paper/2101.01229/) (2021)
- [Action Scene Graphs for Long-Form Understanding of Egocentric Videos](https://mlsanity.com/paper/2312.03391/) (2023)
- [OrphicX: A Causality-Inspired Latent Variable Model for Interpreting Graph Neural Networks](https://mlsanity.com/paper/2203.15209/) (2022)
- [Colar: Effective and Efficient Online Action Detection by Consulting Exemplars](https://mlsanity.com/paper/2203.01057/) (2022)
- [Robust Region Feature Synthesizer for Zero-Shot Object Detection](https://mlsanity.com/paper/2201.00103/) (2022)
- [Encoder-Decoder Architecture for Supervised Dynamic Graph Learning: A Survey](https://mlsanity.com/paper/2203.10480/) (2022)
- [Decoupled Graph Neural Networks for Large Dynamic Graphs](https://mlsanity.com/paper/2305.08273/) (2023)
- [Uncertainty-Guided Appearance-Motion Association Network for Out-of-Distribution Action Detection](https://mlsanity.com/paper/2409.09953/) (2024)
- [Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Models](https://mlsanity.com/paper/2410.14248/) (2024)
- [Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions](https://mlsanity.com/paper/2110.03562/) (2021)
- [Entry-Flipped Transformer for Inference and Prediction of Participant Behavior](https://mlsanity.com/paper/2207.06235/) (2022)
- [Discovering A Variety of Objects in Spatio-Temporal Human-Object Interactions](https://mlsanity.com/paper/2211.07501/) (2022)
- [Interacted Object Grounding in Spatio-Temporal Human-Object Interactions](https://mlsanity.com/paper/2412.19542/) (2024)
- [Explaining Vision and Language through Graphs of Events in Space and Time](https://mlsanity.com/paper/2309.08612/) (2023)
- [Rich Action-semantic Consistent Knowledge for Early Action Prediction](https://mlsanity.com/paper/2201.09169/) (2022)
- [EgoSpeed-Net: Forecasting Speed-Control in Driver Behavior from Egocentric Video Data](https://mlsanity.com/paper/2209.13459/) (2022)
- [Contactless Oxygen Monitoring with Gated Transformer](https://mlsanity.com/paper/2212.03357/) (2022)
- [Spatial Parsing and Dynamic Temporal Pooling networks for Human-Object Interaction detection](https://mlsanity.com/paper/2206.03061/) (2022)
- [GEST: the Graph of Events in Space and Time as a Common Representation between Vision and Language](https://mlsanity.com/paper/2305.12940/) (2023)
- [Interpretable Dynamic Graph Neural Networks for Small Occluded Object Detection and Tracking](https://mlsanity.com/paper/2411.17251/) (2024)
- [Language-guided Recursive Spatiotemporal Graph Modeling for Video Summarization](https://mlsanity.com/paper/2509.05604/) (2025)

## Abstract (arXiv)

In this paper, we investigate a weakly-supervised object detection framework. Most existing frameworks focus on using static images to learn object detectors. However, these detectors often fail to generalize to videos because of the existing domain shift. Therefore, we investigate learning these detectors directly from boring videos of daily activities. Instead of using bounding boxes, we explore the use of action descriptions as supervision since they are relatively easy to gather. A common issue, however, is that objects of interest that are not involved in human actions are often absent in global action descriptions known as "missing label". To tackle this problem, we propose a novel temporal dynamic graph Long Short-Term Memory network (TD-Graph LSTM). TD-Graph LSTM enables global temporal reasoning by constructing a dynamic graph that is based on temporal correlations of object proposals and spans the entire video. The missing label issue for each individual frame can thus be significantly alleviated by transferring knowledge across correlated objects proposals in the whole video. Extensive evaluations on a large-scale daily-life action dataset (i.e., Charades) demonstrates the superiority of our proposed method. We also release object bounding-box annotations for more than 5,000 frames in Charades. We believe this annotated data can also benefit other research on video-based object recognition in the future.