# Predicting Human Activities Using Stochastic Grammar

arXiv: https://arxiv.org/abs/1708.00945 · HTML: https://mlsanity.com/paper/1708.00945/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2017.

## Built on by (most cited first, 22 of 22)

- [3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding](https://mlsanity.com/paper/2103.16397/) (2021)
- [Open-Vocabulary Affordance Detection in 3D Point Clouds](https://mlsanity.com/paper/2303.02401/) (2023)
- [One-Shot Object Affordance Detection in the Wild](https://mlsanity.com/paper/2108.03658/) (2021)
- [Anticipating human actions by correlating past with the future with Jaccard similarity measures](https://mlsanity.com/paper/2105.12414/) (2021)
- [Self-Regulated Learning for Egocentric Video Activity Anticipation](https://mlsanity.com/paper/2111.11631/) (2021)
- [Learning Visual Affordance Grounding from Demonstration Videos](https://mlsanity.com/paper/2108.05675/) (2021)
- [Phrase-Based Affordance Detection via Cyclic Bilateral Interaction](https://mlsanity.com/paper/2202.12076/) (2022)
- [Learning Asynchronous and Sparse Human-Object Interaction in Videos](https://mlsanity.com/paper/2103.02758/) (2021)
- [A Survey on Deep Learning Techniques for Action Anticipation](https://mlsanity.com/paper/2309.17257/) (2023)
- [Human Action Anticipation: A Survey](https://mlsanity.com/paper/2410.14045/) (2024)
- [Activity Grammars for Temporal Action Segmentation](https://mlsanity.com/paper/2312.04266/) (2023)
- [Uncertainty-boosted Robust Video Activity Anticipation](https://mlsanity.com/paper/2404.18648/) (2024)
- [Learning to Discriminate Information for Online Action Detection: Analysis and Application](https://mlsanity.com/paper/2109.03393/) (2021)
- [A large scale multi-view RGBD visual affordance learning dataset](https://mlsanity.com/paper/2203.14092/) (2022)
- [Weakly-Supervised Dense Action Anticipation](https://mlsanity.com/paper/2111.07593/) (2021)
- [Object-agnostic Affordance Categorization via Unsupervised Learning of Graph Embeddings](https://mlsanity.com/paper/2304.05989/) (2023)
- [Visual-Geometric Collaborative Guidance for Affordance Learning](https://mlsanity.com/paper/2410.11363/) (2024)
- [Compositional Benchmark Synthesis for Hierarchical Human Action Recognition](https://mlsanity.com/paper/2608.10765/) (2026)
- [PLAN-B: Predicting Likely Alternative Next Best Sequences for Action Prediction](https://mlsanity.com/paper/2103.15987/) (2021)
- [Explainable Artificial Intelligence for Quantifying Interfering and High-Risk Behaviors in Autism Spectrum Disorder in a Real-World Classroom Environment Using Privacy-Preserving Video Analysis](https://mlsanity.com/paper/2407.21691/) (2024)
- [Capturing Fine-Grained Alignments Improves 3D Affordance Detection](https://mlsanity.com/paper/2506.19312/) (2025)
- [Improving Temporal Action Segmentation via Constraint-Aware Decoding](https://mlsanity.com/paper/2605.10149/) (2026)

## Abstract (arXiv)

This paper presents a novel method to predict future human activities from partially observed RGB-D videos. Human activity prediction is generally difficult due to its non-Markovian property and the rich context between human and environments. We use a stochastic grammar model to capture the compositional structure of events, integrating human actions, objects, and their affordances. We represent the event by a spatial-temporal And-Or graph (ST-AOG). The ST-AOG is composed of a temporal stochastic grammar defined on sub-activities, and spatial graphs representing sub-activities that consist of human actions, objects, and their affordances. Future sub-activities are predicted using the temporal grammar and Earley parsing algorithm. The corresponding action, object, and affordance labels are then inferred accordingly. Extensive experiments are conducted to show the effectiveness of our model on both semantic event parsing and future activity prediction.