# Relational Action Forecasting

arXiv: https://arxiv.org/abs/1904.04231 · HTML: https://mlsanity.com/paper/1904.04231/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2019.

## Built on by (most cited first, 27 of 27)

- [LaneRCNN: Distributed Representations for Graph-Centric Motion Forecasting](https://mlsanity.com/paper/2101.06653/) (2021)
- [AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?](https://mlsanity.com/paper/2307.16368/) (2023)
- [On the Benefits of 3D Pose and Tracking for Human Action Recognition](https://mlsanity.com/paper/2304.01199/) (2023)
- [Visual Abductive Reasoning](https://mlsanity.com/paper/2203.14040/) (2022)
- [Rethinking Learning Approaches for Long-Term Action Anticipation](https://mlsanity.com/paper/2210.11566/) (2022)
- [Object-centric Video Representation for Long-term Action Anticipation](https://mlsanity.com/paper/2311.00180/) (2023)
- [Composable Augmentation Encoding for Video Representation Learning](https://mlsanity.com/paper/2104.00616/) (2021)
- [A Survey on Deep Learning Techniques for Action Anticipation](https://mlsanity.com/paper/2309.17257/) (2023)
- [A Hierarchical Variational Neural Uncertainty Model for Stochastic Video Prediction](https://mlsanity.com/paper/2110.03446/) (2021)
- [Instance-Level Relative Saliency Ranking with Graph Reasoning](https://mlsanity.com/paper/2107.03824/) (2021)
- [Dyadic Human Motion Prediction](https://mlsanity.com/paper/2112.00396/) (2021)
- [Aware of the History: Trajectory Forecasting with the Local Behavior Data](https://mlsanity.com/paper/2207.09646/) (2022)
- [Human Action Anticipation: A Survey](https://mlsanity.com/paper/2410.14045/) (2024)
- [Action Anticipation from SoccerNet Football Video Broadcasts](https://mlsanity.com/paper/2504.12021/) (2025)
- [Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection](https://mlsanity.com/paper/2411.10922/) (2024)
- [Smart Director: An Event-Driven Directing System for Live Broadcasting](https://mlsanity.com/paper/2201.04024/) (2022)
- [Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction](https://mlsanity.com/paper/2107.05140/) (2021)
- [Reasoning Graph Networks for Kinship Verification: from Star-shaped to Hierarchical](https://mlsanity.com/paper/2109.02219/) (2021)
- [Rich Action-semantic Consistent Knowledge for Early Action Prediction](https://mlsanity.com/paper/2201.09169/) (2022)
- [Context-Aware Human Behavior Prediction Using Multimodal Large Language Models: Challenges and Insights](https://mlsanity.com/paper/2504.00839/) (2025)
- [Poly-Autoregressive Prediction for Modeling Interactions](https://mlsanity.com/paper/2502.08646/) (2025)
- [Seeing is Believing (and Predicting): Context-Aware Multi-Human Behavior Prediction with Vision Language Models](https://mlsanity.com/paper/2512.15957/) (2025)
- [Understanding in Artificial Intelligence](https://mlsanity.com/paper/2101.06573/) (2021)
- [Discovering Multi-Label Actor-Action Association in a Weakly Supervised Setting](https://mlsanity.com/paper/2101.08567/) (2021)
- [Adaptive Recursive Circle Framework for Fine-grained Action Recognition](https://mlsanity.com/paper/2107.11813/) (2021)
- [Group Activity Recognition in Computer Vision: A Comprehensive Review, Challenges, and Future Perspectives](https://mlsanity.com/paper/2307.13541/) (2023)
- [HINT-Plan: Human Intention-Aware Robot Task Planning in Context-Rich Environments using Vision Language Models](https://mlsanity.com/paper/2609.17771/) (2026)

## Abstract (arXiv)

This paper focuses on multi-person action forecasting in videos. More precisely, given a history of H previous frames, the goal is to detect actors and to predict their future actions for the next T frames. Our approach jointly models temporal and spatial interactions among different actors by constructing a recurrent graph, using actor proposals obtained with Faster R-CNN as nodes. Our method learns to select a subset of discriminative relations without requiring explicit supervision, thus enabling us to tackle challenging visual data. We refer to our model as Discriminative Relational Recurrent Network (DRRN). Evaluation of action prediction on AVA demonstrates the effectiveness of our proposed method compared to simpler baselines. Furthermore, we significantly improve performance on the task of early action classification on J-HMDB, from the previous SOTA of 48% to 60%.