mlsanityData to 7 Oct 2026

Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation

arXiv 2210.12649 ↗First posted 23 Oct 2022cs.CV cs.ROcited by 41

Not on the map. Why?

Abstract (arXiv)

Although human action anticipation is a task which is inherently multi-modal, state-of-the-art methods on well known action anticipation datasets leverage this data by applying ensemble methods and averaging scores of unimodal anticipation networks. In this work we introduce transformer based modality fusion techniques, which unify multi-modal data at an early stage. Our Anticipative Feature Fusion Transformer (AFFT) proves to be superior to popular score fusion approaches and presents state-of-the-art results outperforming previous methods on EpicKitchens-100 and EGTEA Gaze+. Our model is easily extensible and allows for adding new modalities without architectural changes. Consequently, we extracted audio features on EpicKitchens-100 which we add to the set of commonly used features in the community.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 42 of its 53 references that are in the map. First posted on arXiv 23 Oct 2022.

Builds on

42 of 53 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  9. 2021Video Swin Transformercited by 1,090
  10. 2016Deep Networks with Stochastic Depthcited by 973
  11. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  12. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
30 more
  1. 2020Quantifying Attention Flow in Transformerscited by 759
  2. 2017Look, Listen and Learncited by 452
  3. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  4. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  5. 2017Objects that Soundcited by 254
  6. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  7. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  8. 2019EPIC-Fusion: Audio-Visual Temporal Binding for Egocentric Action Recognitioncited by 226
  9. 2020The EPIC-KITCHENS Dataset: Collection, Challenges and Baselinescited by 221
  10. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  11. 2021Anticipative Video Transformercited by 197
  12. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  13. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  14. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 160
  15. 2019SCSampler: Sampling Salient Clips from Video for Efficient Action Recognitioncited by 133
  16. 2015Visually Indicated Soundscited by 128
  17. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  18. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 106
  19. 2019What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attentioncited by 104
  20. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  21. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  22. 2017RED: Reinforced Encoder-Decoder Networks for Action Anticipationcited by 77
  23. 2017ActionVLAD: Learning spatio-temporal aggregation for action classificationcited by 73
  24. 2017Learnable pooling with Context Gating for video classificationcited by 58
  25. 2022Future Transformer for Long-term Action Anticipationcited by 57
  26. 2019Next-Active-Object prediction from Egocentric Videoscited by 55
  27. 2021With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognitioncited by 41
  28. 2021Learning to Anticipate Egocentric Actions by Imaginationcited by 40
  29. 2019Learning to Localize Sound Sources in Visual Scenes: Analysis and Applicationscited by 31
  30. 2021Multi-Modal Temporal Convolutional Network for Anticipating Actions in Egocentric Videoscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 14; 2025: 10; 2026: 6.

41 papers.

Sort:

By area:

  1. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  2. 2023An Outlook into the Future of Egocentric Visioncited by 84
  3. 2025Back to the Features: DINO as a Foundation for Video World Modelscited by 56
  4. 2024DINO-Foresight: Looking into the Future with DINOcited by 44
  5. 2022Interaction Region Visual Transformer for Egocentric Action Anticipationcited by 36
  6. 2024Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Modelscited by 32
  7. 2023StillFast: An End-to-End Approach for Short-Term Object Interaction Anticipationcited by 23
  8. 2023Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Reviewcited by 20
  9. 2023Leveraging Next-Active Objects for Context-Aware Anticipation in Egocentric Videoscited by 16
  10. 2024AFF-ttention! Affordances and Attention models for Short-Term Object Interaction Anticipationcited by 14
  11. 2023A Survey on Deep Learning Techniques for Action Anticipationcited by 11
  12. 2025Multimodal Large Models Are Effective Action Anticipatorscited by 10
29 more
  1. 2023GePSAn: Generative Procedure Step Anticipation in Cooking Videoscited by 9
  2. 2023DiffAnt: Diffusion Models for Action Anticipationcited by 9
  3. 2024A Backpack Full of Skills: Egocentric Video Understanding with Diverse Task Perspectivescited by 9
  4. 2024Semantically Guided Representation Learning For Action Anticipationcited by 9
  5. 2025Action Anticipation from SoccerNet Football Video Broadcastscited by 9
  6. 2024Human Action Anticipation: A Surveycited by 8
  7. 2024On the Efficacy of Text-Based Input Modalities for Action Anticipationcited by 6
  8. 2024Gated Temporal Diffusion for Stochastic Long-Term Dense Anticipationcited by 6
  9. 2024CathAction: A Benchmark for Endovascular Intervention Understandingcited by 5
  10. 2024Mamba Fusion: Learning Actions Through Questioningcited by 5
  11. 2025Hier-EgoPack: Hierarchical Egocentric Video Understanding with Diverse Task Perspectivescited by 5
  12. 2025Vision and Intention Boost Large Language Model in Long-Term Action Anticipationcited by 5
  13. 2023VS-TransGRU: A Novel Transformer-GRU-based Framework Enhanced by Visual-Semantic Fusion for Egocentric Action Anticipationcited by 4
  14. 2023Robust Audio Anti-Spoofing with Fusion-Reconstruction Learning on Multi-Order Spectrogramscited by 4
  15. 2025Uni-Hand: Universal Hand Motion Forecasting in Egocentric Viewscited by 4
  16. 2024Multi-modal Crowd Counting via Modal Emulationcited by 3
  17. 2024Muscles in Time: Learning to Understand Human Motion by Simulating Muscle Activationscited by 3
  18. 2025MANTA: Diffusion Mamba for Efficient and Effective Stochastic Long-Term Dense Anticipationcited by 3
  19. 2026Action-Guided Attention for Video Action Anticipationcited by 3
  20. 2023MULTISCRIPT: Multimodal Script Learning for Supporting Open Domain Everyday Taskscited by 2
  21. 2024From Recognition to Prediction: Leveraging Sequence Reasoning for Action Anticipationcited by 2
  22. 2025Multi-level and Multi-modal Action Anticipationcited by 2
  23. 2026IMPACT-CYCLE: A Contract-Based Multi-Agent System for Claim-Level Supervisory Correction of Long-Video Semantic Memorycited by 2
  24. 2026IMPACT-HOI: Supervisory Control for Onset-Anchored Partial HOI Event Constructioncited by 2
  25. 2025MixANT: Observation-dependent Memory Propagation for Stochastic Dense Action Anticipationcited by 1
  26. 2025Countering Multi-modal Representation Collapse through Rank-targeted Fusioncited by 1
  27. 2026Integrating Affordances and Attention models for Short-Term Object Interaction Anticipationcited by 1
  28. 2026SAGE: Synchronized Action-Gaze Recognition and Anticipation for Human Behavior Understandingcited by 0
  29. 2026Neuro-Symbolic Hierarchical Intention Anticipation in Human Behaviorcited by 0