mlsanityData to 14 Oct 2026

Future Transformer for Long-term Action Anticipation

arXiv 2205.14022 ↗First posted 27 May 2022cs.CVcited by 58

Not on the map. Why?

Abstract (arXiv)

The task of predicting future actions from a video is crucial for a real-world agent interacting with others. When anticipating actions in the distant future, we humans typically consider long-term relations over the whole sequence of actions, i.e., not only observed actions in the past but also potential actions in the future. In a similar spirit, we propose an end-to-end attention model for action anticipation, dubbed Future Transformer (FUTR), that leverages global attention over all input frames and output tokens to predict a minutes-long sequence of future actions. Unlike the previous autoregressive models, the proposed method learns to predict the whole sequence of future actions in parallel decoding, enabling more accurate and fast inference for long-term anticipation. We evaluate our method on two standard benchmarks for long-term action anticipation, Breakfast and 50 Salads, achieving state-of-the-art results.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 50 of its 59 references that are in the map. First posted on arXiv 27 May 2022.

Builds on

50 of 59 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2020End-to-End Object Detection with Transformerscited by 7,414
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  8. 2015Fast R-CNNcited by 2,933
  9. 2017Non-local Neural Networkscited by 2,293
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  11. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
38 more
  1. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  2. 2021ViViT: A Video Vision Transformercited by 1,620
  3. 2020Rethinking Attention with Performerscited by 1,601
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  5. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  6. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  7. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  8. 2021Multiscale Vision Transformerscited by 897
  9. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  10. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  11. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  12. 2019Stand-Alone Self-Attention in Vision Modelscited by 542
  13. 2020Rescaling Egocentric Visioncited by 540
  14. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 508
  15. 2017Non-Autoregressive Neural Machine Translationcited by 463
  16. 2019MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentationcited by 373
  17. 2019Attention Augmented Convolutional Networkscited by 363
  18. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  19. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  20. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  21. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  22. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  23. 2021Anticipative Video Transformercited by 200
  24. 2021ASFormer: Transformer for Action Segmentationcited by 168
  25. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  26. 2021VidTr: Video Transformer Without Convolutionscited by 112
  27. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 107
  28. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 105
  29. 2020Alleviating Over-segmentation Errors by Detecting Action Boundariescited by 87
  30. 2017Weakly Supervised Action Learning with RNN based Fine-to-coarse Modelingcited by 70
  31. 2021Activity Graph Transformer for Temporal Action Localizationcited by 48
  32. 2018Zero-Shot Anticipation for Instructional Activitiescited by 43
  33. 2019Predicting the Future: A Jointly Learnt Model for Action Anticipationcited by 41
  34. 2020Long-Term Anticipation of Activities with Cycle Consistencycited by 40
  35. 2019Uncertainty-Aware Anticipation of Activitiescited by 39
  36. 2021Anticipating human actions by correlating past with the future with Jaccard similarity measurescited by 36
  37. 2021CTRL-C: Camera calibration TRansformer with Line-Classificationcited by 33
  38. 2021Relational Self-Attention: What's Missing in Attention for Video Understandingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 15; 2024: 12; 2025: 18; 2026: 7.

58 papers.

Sort:

By area:

  1. 2023A New Comprehensive Benchmark for Semi-supervised Video Anomaly Detection and Anticipationcited by 42
  2. 2022Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipationcited by 41
  3. 2023Motion Flow Matching for Human Motion Synthesis and Editingcited by 38
  4. 2022Interaction Region Visual Transformer for Egocentric Action Anticipationcited by 37
  5. 2024Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Modelscited by 33
  6. 2022Vision Transformers for Action Recognition: A Surveycited by 25
  7. 2023Object-centric Video Representation for Long-term Action Anticipationcited by 24
  8. 2023StillFast: An End-to-End Approach for Short-Term Object Interaction Anticipationcited by 23
  9. 2023Event-Guided Procedure Planning from Instructional Videos with Text Supervisioncited by 21
  10. 2023HA-ViD: A Human Assembly Video Dataset for Comprehensive Assembly Knowledge Understandingcited by 18
  11. 2022Predicting the Next Action by Modeling the Abstract Goalcited by 14
  12. 2023Action Anticipation with Goal Consistencycited by 12
38 more
  1. 2025Modeling Multiple Normal Action Representations for Error Detection in Procedural Taskscited by 12
  2. 2023A Survey on Deep Learning Techniques for Action Anticipationcited by 11
  3. 2024PEAR: Phrase-Based Hand-Object Interaction Anticipationcited by 10
  4. 2025Multimodal Large Models Are Effective Action Anticipatorscited by 10
  5. 2023GePSAn: Generative Procedure Step Anticipation in Cooking Videoscited by 9
  6. 2023DiffAnt: Diffusion Models for Action Anticipationcited by 9
  7. 2024Human Action Anticipation: A Surveycited by 9
  8. 2025Action Anticipation from SoccerNet Football Video Broadcastscited by 9
  9. 2023Transformer-Based Autonomous Driving Models and Deployment-Oriented Compression: A Surveycited by 8
  10. 2024ActionDiffusion: An Action-aware Diffusion Model for Procedure Planning in Instructional Videoscited by 8
  11. 2023JOADAA: joint online action detection and action anticipationcited by 7
  12. 2024Uncertainty-boosted Robust Video Activity Anticipationcited by 7
  13. 2025Hierarchical and Multimodal Data for Daily Activity Understandingcited by 7
  14. 2023Multiscale Video Pretraining for Long-Term Activity Forecastingcited by 6
  15. 2024Bidirectional Progressive Transformer for Interaction Intention Anticipationcited by 6
  16. 2024Gated Temporal Diffusion for Stochastic Long-Term Dense Anticipationcited by 6
  17. 2025CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentationcited by 6
  18. 2022FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observationscited by 5
  19. 2025Vision and Intention Boost Large Language Model in Long-Term Action Anticipationcited by 5
  20. 2025Online Generic Event Boundary Detectioncited by 5
  21. 2023VS-TransGRU: A Novel Transformer-GRU-based Framework Enhanced by Visual-Semantic Fusion for Egocentric Action Anticipationcited by 4
  22. 2023Knowledge-Guided Short-Context Action Anticipation in Human-Centric Videoscited by 4
  23. 2024A Survey of Vision Transformers in Autonomous Driving: Current Trends and Future Directionscited by 4
  24. 2024QueryMamba: A Mamba-Based Encoder-Decoder Architecture with a Statistical Verb-Noun Interaction Module for Video Action Forecasting @ Ego4D Long-Term Action Anticipation Challenge 2024cited by 4
  25. 2024ActFusion: a Unified Diffusion Model for Action Segmentation and Anticipationcited by 4
  26. 2024HiMemFormer: Hierarchical Memory-Aware Transformer for Multi-Agent Action Anticipationcited by 3
  27. 2025MANTA: Diffusion Mamba for Efficient and Effective Stochastic Long-Term Dense Anticipationcited by 3
  28. 2025CLAD: Constrained Latent Action Diffusion for Vision-Language Procedure Planningcited by 2
  29. 2025Multi-level and Multi-modal Action Anticipationcited by 2
  30. 2025Bidirectional Action Sequence Learning for Long-term Action Anticipation with Large Language Modelscited by 2
  31. 2026Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistencycited by 2
  32. 2022Inferring Past Human Actions in Homes with Abductive Reasoningcited by 1
  33. 2025FDSG: Forecasting Dynamic Scene Graphscited by 1
  34. 2025MixANT: Observation-dependent Memory Propagation for Stochastic Dense Action Anticipationcited by 1
  35. 2025Countering Multi-modal Representation Collapse through Rank-targeted Fusioncited by 1
  36. 2025ProcObject-10K: Benchmarking Object-Centric Procedural Understanding in Instructional Videoscited by 1
  37. 2026EggHand: A Multimodal Foundation Model for Egocentric Hand Pose Forecastingcited by 1
  38. 2026FactCheck: Feasibility-aware Long-term Action Anticipation with Multi-agent Collaborationcited by 1

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipation
  2. 2023AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?
  3. 2024Temporal Context Consistency Above All: Enhancing Long-Term Anticipation by Learning and Enforcing Temporal Constraints
  4. 2026The Right Future for Action: Learning Action-Relevant Predictive States in World Action Models
  5. 2026Foresight Without Seeing: Latent Futures for World Action Models
  6. 2026Privileged Foresight Distillation: Zero-Cost Future Correction for World Action Models
  7. 2026PACT-WAM: Predicting Actions and Visual Foresight with Compact Temporal Encoding for Robot Manipulation
  8. 2023Palm: Predicting Actions through Language Models @ Ego4D Long-Term Action Anticipation Challenge 2023