mlsanityData to 14 Oct 2026

DiffAnt: Diffusion Models for Action Anticipation

arXiv 2311.15991 ↗First posted 27 Nov 2023cs.CVcited by 9

On the map: Diffusion models → Video generation

Abstract (arXiv)

Anticipating future actions is inherently uncertain. Given an observed video segment containing ongoing actions, multiple subsequent actions can plausibly follow. This uncertainty becomes even larger when predicting far into the future. However, the majority of existing action anticipation models adhere to a deterministic approach, neglecting to account for future uncertainties. In this work, we rethink action anticipation from a generative view, employing diffusion models to capture different possible future actions. In this framework, future actions are iteratively generated from standard Gaussian noise in the latent space, conditioned on the observed video, and subsequently transitioned into the action space. Extensive experiments on four benchmark datasets, i.e., Breakfast, 50Salads, EpicKitchens, and EGTEA Gaze+, are performed and the proposed method achieves superior or comparable results to state-of-the-art methods, showing the effectiveness of a generative approach for action anticipation. Our code and trained models will be published on GitHub.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 53 of its 63 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

53 of 63 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Diffusion Models for Video Prediction and Infilling15 Jun
  2. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation19 May
  3. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Denoising Diffusion Implicit Modelscited by 9,781
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  11. 2013Auto-Encoding Variational Bayescited by 4,053
  12. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
38 more
  1. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  4. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  5. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  6. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  7. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  8. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 499
  9. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 450
  10. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 373
  11. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  12. 2022Understanding Diffusion Models: A Unified Perspectivecited by 340
  13. 2021SegDiff: Image Segmentation with Diffusion Probabilistic Modelscited by 280
  14. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 225
  15. 2021Anticipative Video Transformercited by 200
  16. 2021ASFormer: Transformer for Action Segmentationcited by 168
  17. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  18. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 107
  19. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 105
  20. 2018Timeception for Complex Action Recognitioncited by 89
  21. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  22. 2017RED: Reinforced Encoder-Decoder Networks for Action Anticipationcited by 78
  23. 2023Diffusion Action Segmentationcited by 74
  24. 2017ActionVLAD: Learning spatio-temporal aggregation for action classificationcited by 73
  25. 2017Weakly Supervised Action Learning with RNN based Fine-to-coarse Modelingcited by 70
  26. 2022Latent Diffusion Energy-Based Model for Interpretable Text Modelingcited by 67
  27. 2022BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesiscited by 64
  28. 2022Future Transformer for Long-term Action Anticipationcited by 58
  29. 2019VideoGraph: Recognizing Minutes-Long Human Activities in Videoscited by 57
  30. 2022BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesiscited by 44
  31. 2022Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipationcited by 41
  32. 2020Long-Term Anticipation of Activities with Cycle Consistencycited by 40
  33. 2022Intention-Conditioned Long-Term Human Egocentric Action Forecastingcited by 35
  34. 2019A Variational Auto-Encoder Model for Stochastic Point Processescited by 30
  35. 2022Rethinking Learning Approaches for Long-Term Action Anticipationcited by 29
  36. 2020Adversarial Generative Grammars for Human Activity Predictioncited by 19
  37. 2023A Survey on Deep Learning Techniques for Action Anticipationcited by 11
  38. 2022A-ACT: Action Anticipation through Cycle Transformationscited by 6

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 4.

9 papers.

  1. 2024MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videoscited by 18
  2. 2024Human Action Anticipation: A Surveycited by 9
  3. 2025Action Anticipation from SoccerNet Football Video Broadcastscited by 9
  4. 2024Uncertainty-boosted Robust Video Activity Anticipationcited by 7
  5. 2024QueryMamba: A Mamba-Based Encoder-Decoder Architecture with a Statistical Verb-Noun Interaction Module for Video Action Forecasting @ Ego4D Long-Term Action Anticipation Challenge 2024cited by 4
  6. 2025MANTA: Diffusion Mamba for Efficient and Effective Stochastic Long-Term Dense Anticipationcited by 3
  7. 2025MixANT: Observation-dependent Memory Propagation for Stochastic Dense Action Anticipationcited by 1
  8. 2024Temporal Context Consistency Above All: Enhancing Long-Term Anticipation by Learning and Enforcing Temporal Constraintscited by 0
  9. 2025See It Before You Grab It: Deep Learning-based Action Anticipation in Basketballcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Foresight Without Seeing: Latent Futures for World Action Models
  2. 2025Multi-level and Multi-modal Action Anticipation
  3. 2026Privileged Foresight Distillation: Zero-Cost Future Correction for World Action Models
  4. 2026The Right Future for Action: Learning Action-Relevant Predictive States in World Action Models
  5. 2026NoiseGate: Learning Per-Latent Timestep Schedules as Information Gating in World Action Models
  6. 2024From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation
  7. 2024ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipation
  8. 2025Action Anticipation at a Glimpse: To What Extent Can Multimodal Cues Replace Video?