# Modular Action Concept Grounding in Semantic Video Prediction

arXiv: https://arxiv.org/abs/2011.11201 · HTML: https://mlsanity.com/paper/2011.11201/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2020.

## Built on by (most cited first, 12 of 12)

- [SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models](https://mlsanity.com/paper/2210.05861/) (2022)
- [Towards Scene Graph Anticipation](https://mlsanity.com/paper/2403.04899/) (2024)
- [SSGVS: Semantic Scene Graph-to-Video Synthesis](https://mlsanity.com/paper/2211.06119/) (2022)
- [See, Plan, Predict: Language-guided Cognitive Planning with Video Prediction](https://mlsanity.com/paper/2210.03825/) (2022)
- [Dreamweaver: Learning Compositional World Models from Pixels](https://mlsanity.com/paper/2501.14174/) (2025)
- [Video Generation Beyond a Single Clip](https://mlsanity.com/paper/2304.07483/) (2023)
- [Animating Street View](https://mlsanity.com/paper/2310.08534/) (2023)
- [Fast Fourier Inception Networks for Occluded Video Prediction](https://mlsanity.com/paper/2306.10346/) (2023)
- [Learning Disentangled Representation in Object-Centric Models for Visual Dynamics Prediction via Transformers](https://mlsanity.com/paper/2407.03216/) (2024)
- [Pair-wise Layer Attention with Spatial Masking for Video Prediction](https://mlsanity.com/paper/2311.11289/) (2023)
- [InterOCF: Spatio-Temporal 2D-3D Interaction for Camera-Only 4D Occupancy Forecasting](https://mlsanity.com/paper/2607.24431/) (2026)
- [Valerant: An Automatic Navigable Game Map Generator via Action-Conditioned World Model Exploration](https://mlsanity.com/paper/2609.09418/) (2026)

## Abstract (arXiv)

Recent works in video prediction have mainly focused on passive forecasting and low-level action-conditional prediction, which sidesteps the learning of interaction between agents and objects. We introduce the task of semantic action-conditional video prediction, which uses semantic action labels to describe those interactions and can be regarded as an inverse problem of action recognition. The challenge of this new task primarily lies in how to effectively inform the model of semantic action information. Inspired by the idea of Mixture of Experts, we embody each abstract label by a structured combination of various visual concept learners and propose a novel video prediction model, Modular Action Concept Network (MAC). Our method is evaluated on two newly designed synthetic datasets, CLEVR-Building-Blocks and Sapien-Kitchen, and one real-world dataset called Tower-Creation. Extensive experiments demonstrate that MAC can correctly condition on given instructions and generate corresponding future frames without need of bounding boxes. We further show that the trained model can make out-of-distribution generalization, be quickly adapted to new object categories and exploit its learnt features for object detection, showing the progression towards higher-level cognitive abilities. More visualizations can be found at http://www.pair.toronto.edu/mac/.