mlsanityData to 7 Oct 2026

Compositional Video Synthesis with Action Graphs

arXiv 2006.15327 ↗First posted 27 Jun 2020cs.CV cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

Videos of actions are complex signals containing rich compositional structure in space and time. Current video generation methods lack the ability to condition the generation on multiple coordinated and potentially simultaneous timed actions. To address this challenge, we propose to represent the actions in a graph structure called Action Graph and present the new ``Action Graph To Video'' synthesis task. Our generative model for this task (AG2Vid) disentangles motion and appearance features, and by incorporating a scheduling mechanism for actions facilitates a timely and coordinated video generation. We train and evaluate AG2Vid on the CATER and Something-Something V2 datasets, and show that the resulting videos have better visual quality and semantic consistency compared to baselines. Finally, our model demonstrates zero-shot abilities by synthesizing novel compositions of the learned actions. For code and pretrained models, see the project page https://roeiherz.github.io/AG2Video

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 10; 2023: 7; 2024: 9; 2025: 2; 2026: 1.

34 papers.

Sort:

By area:

  1. 2024Navigation World Modelscited by 262
  2. 2021Conditional Object-Centric Learning from Videocited by 247
  3. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  4. 2021Illiterate DALL-E Learns to Composecited by 147
  5. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 101
  6. 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learningcited by 87
  7. 2021Object-Region Video Transformerscited by 73
  8. 2021Learning to Compose Visual Relationscited by 72
  9. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  10. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  11. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learningcited by 66
  12. 2024TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generationcited by 42
22 more
  1. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  2. 2022(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answeringcited by 29
  3. 2021Compositional Transformers for Scene Generationcited by 24
  4. 2024TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answeringcited by 24
  5. 2022PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Datacited by 14
  6. 2024Compositional Video Generation as Flow Equalizationcited by 12
  7. 2024Towards Scene Graph Anticipationcited by 10
  8. 2024VISAGE: Video Synthesis using Action Graphs for Surgerycited by 9
  9. 2022SSGVS: Semantic Scene Graph-to-Video Synthesiscited by 8
  10. 2023Benchmarks for Physical Reasoning AIcited by 8
  11. 20233VL: Using Trees to Improve Vision-Language Models' Interpretabilitycited by 7
  12. 2022WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Predictioncited by 5
  13. 2022FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observationscited by 5
  14. 2022Patch-based Object-centric Transformers for Efficient Video Generationcited by 3
  15. 2023Video Generation Beyond a Single Clipcited by 3
  16. 2025Data-Driven Loss Functions for Inference-Time Optimization in Text-to-Imagecited by 3
  17. 2023Language-free Compositional Action Generation via Decoupling Refinementcited by 2
  18. 2022Bringing Image Scene Structure to Video via Frame-Clip Consistency of Object Tokenscited by 0
  19. 2024Object-Attribute-Relation Representation Based Video Semantic Communicationcited by 0
  20. 2024Box2Flow: Instance-based Action Flow Graphs from Videoscited by 0
  21. 2025AI Powered High Quality Text to Video Generation with Enhanced Temporal Consistencycited by 0
  22. 2026GraphVid: Interactive Graph-Controllable Video Generationcited by 0