# SlowFast Rolling-Unrolling LSTMs for Action Anticipation in Egocentric Videos

arXiv: https://arxiv.org/abs/2109.00829 · HTML: https://mlsanity.com/paper/2109.00829/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 2 Sep 2021.

## Builds on (most cited first, 25 of 25)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [Beyond Short Snippets: Deep Networks for Video Classification](https://mlsanity.com/paper/1503.08909/) (2015)
- [The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines](https://mlsanity.com/paper/2005.00343/) (2020)
- [Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video](https://mlsanity.com/paper/2005.02190/) (2020)
- [Multi-Modal Domain Adaptation for Fine-Grained Action Recognition](https://mlsanity.com/paper/2001.09691/) (2020)
- [Peeking into the Future: Predicting Future Person Activities and Locations in Videos](https://mlsanity.com/paper/1902.03748/) (2019)
- [When will you do what? - Anticipating Temporal Occurrences of Activities](https://mlsanity.com/paper/1804.00892/) (2018)
- [RED: Reinforced Encoder-Decoder Networks for Action Anticipation](https://mlsanity.com/paper/1707.04818/) (2017)
- [Next-Active-Object prediction from Egocentric Videos](https://mlsanity.com/paper/1904.05250/) (2019)
- [First-Person Activity Forecasting with Online Inverse Reinforcement Learning](https://mlsanity.com/paper/1612.07796/) (2016)
- [Symbiotic Attention with Privileged Information for Egocentric Action Recognition](https://mlsanity.com/paper/2002.03137/) (2020)
- [Knowledge Distillation for Action Anticipation via Label Smoothing](https://mlsanity.com/paper/2004.07711/) (2020)
- [Modality Compensation Network: Cross-Modal Adaptation for Action Recognition](https://mlsanity.com/paper/2001.11657/) (2020)
- [Visual Forecasting by Imitating Dynamics in Natural Sequences](https://mlsanity.com/paper/1708.05827/) (2017)
- [Anticipation in Human-Robot Cooperation: A Recurrent Neural Network Approach for Multiple Action Sequences Prediction](https://mlsanity.com/paper/1802.10503/) (2018)
- [Forecasting Hands and Objects in Future Frames](https://mlsanity.com/paper/1705.07328/) (2017)
- [Multi-Modal Multi-Scale Deep Learning for Large-Scale Image Annotation](https://mlsanity.com/paper/1709.01220/) (2017)

## Built on by (most cited first, 20 of 20)

- [MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain](https://mlsanity.com/paper/2209.08691/) (2022)
- [Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Models](https://mlsanity.com/paper/2405.20305/) (2024)
- [Rethinking Learning Approaches for Long-Term Action Anticipation](https://mlsanity.com/paper/2210.11566/) (2022)
- [StillFast: An End-to-End Approach for Short-Term Object Interaction Anticipation](https://mlsanity.com/paper/2304.03959/) (2023)
- [A Survey on Deep Learning Techniques for Action Anticipation](https://mlsanity.com/paper/2309.17257/) (2023)
- [Semantically Guided Representation Learning For Action Anticipation](https://mlsanity.com/paper/2407.02309/) (2024)
- [Multimodal Large Models Are Effective Action Anticipators](https://mlsanity.com/paper/2501.00795/) (2025)
- [TAMFormer: Multi-Modal Transformer with Learned Attention Mask for Early Intent Prediction](https://mlsanity.com/paper/2210.14714/) (2022)
- [Human Action Anticipation: A Survey](https://mlsanity.com/paper/2410.14045/) (2024)
- [Unified Recurrence Modeling for Video Action Anticipation](https://mlsanity.com/paper/2206.01009/) (2022)
- [Privacy-preserving Early Detection of Epileptic Seizures in Videos](https://mlsanity.com/paper/2309.08794/) (2023)
- [Learning to Discriminate Information for Online Action Detection: Analysis and Application](https://mlsanity.com/paper/2109.03393/) (2021)
- [VS-TransGRU: A Novel Transformer-GRU-based Framework Enhanced by Visual-Semantic Fusion for Egocentric Action Anticipation](https://mlsanity.com/paper/2307.03918/) (2023)
- [Inductive Attention for Video Action Anticipation](https://mlsanity.com/paper/2212.08830/) (2022)
- [Early Action Recognition with Action Prototypes](https://mlsanity.com/paper/2312.06598/) (2023)
- [Intention Action Anticipation Model with Guide-Feedback Loop Mechanism](https://mlsanity.com/paper/2403.12450/) (2024)
- [FactCheck: Feasibility-aware Long-term Action Anticipation with Multi-agent Collaboration](https://mlsanity.com/paper/2606.14778/) (2026)
- [Action-Dynamics Modeling and Cross-Temporal Interaction for Online Action Understanding](https://mlsanity.com/paper/2510.10682/) (2025)
- [State-Change Learning for Prediction of Future Events in Endoscopic Videos](https://mlsanity.com/paper/2510.12904/) (2025)
- [Neuro-Symbolic Hierarchical Intention Anticipation in Human Behavior](https://mlsanity.com/paper/2609.17064/) (2026)

## Abstract (arXiv)

Action anticipation in egocentric videos is a difficult task due to the inherently multi-modal nature of human actions. Additionally, some actions happen faster or slower than others depending on the actor or surrounding context which could vary each time and lead to different predictions. Based on this idea, we build upon RULSTM architecture, which is specifically designed for anticipating human actions, and propose a novel attention-based technique to evaluate, simultaneously, slow and fast features extracted from three different modalities, namely RGB, optical flow, and extracted objects. Two branches process information at different time scales, i.e., frame-rates, and several fusion schemes are considered to improve prediction accuracy. We perform extensive experiments on EpicKitchens-55 and EGTEA Gaze+ datasets, and demonstrate that our technique systematically improves the results of RULSTM architecture for Top-5 accuracy metric at different anticipation times.