# Learning to Anticipate Future with Dynamic Context Removal

arXiv: https://arxiv.org/abs/2204.02587 · HTML: https://mlsanity.com/paper/2204.02587/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 67 references that are in the map.
- First posted on arXiv 6 Apr 2022.

## Builds on (most cited first, 28 of 28)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [Long-Term Feature Banks for Detailed Video Understanding](https://mlsanity.com/paper/1812.05038/) (2018)
- [Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video](https://mlsanity.com/paper/2005.02190/) (2020)
- [When will you do what? - Anticipating Temporal Occurrences of Activities](https://mlsanity.com/paper/1804.00892/) (2018)
- [RED: Reinforced Encoder-Decoder Networks for Action Anticipation](https://mlsanity.com/paper/1707.04818/) (2017)
- [Next-Active-Object prediction from Egocentric Videos](https://mlsanity.com/paper/1904.05250/) (2019)
- [Forecasting Action through Contact Representations from First Person Video](https://mlsanity.com/paper/2102.00649/) (2021)
- [Recurrent Neural Networks for Driver Activity Anticipation via Sensory-Fusion Architecture](https://mlsanity.com/paper/1509.05016/) (2015)
- [Learning to Anticipate Egocentric Actions by Imagination](https://mlsanity.com/paper/2101.04924/) (2021)
- [Anticipating Visual Representations from Unlabeled Video](https://mlsanity.com/paper/1504.08023/) (2015)
- [Forecasting Human-Object Interaction: Joint Prediction of Motor Attention and Actions in First Person Video](https://mlsanity.com/paper/1911.10967/) (2019)
- [Shuffle and Learn: Unsupervised Learning using Temporal Order Verification](https://mlsanity.com/paper/1603.08561/) (2016)
- [Multiple Instance Curriculum Learning for Weakly Supervised Object Detection](https://mlsanity.com/paper/1711.09191/) (2017)
- [TransAction: ICL-SJTU Submission to EPIC-Kitchens Action Anticipation Challenge 2021](https://mlsanity.com/paper/2107.13259/) (2021)
- [Temporal Aggregate Representations for Long-Range Video Understanding](https://mlsanity.com/paper/2006.00830/) (2020)
- [Autonomous Driving: Framework for Pedestrian Intention Estimationin a Real World Scenario](https://mlsanity.com/paper/2006.02711/) (2020)

## Abstract (arXiv)

Anticipating future events is an essential feature for intelligent systems and embodied AI. However, compared to the traditional recognition task, the uncertainty of future and reasoning ability requirement make the anticipation task very challenging and far beyond solved. In this filed, previous methods usually care more about the model architecture design or but few attention has been put on how to train an anticipation model with a proper learning policy. To this end, in this work, we propose a novel training scheme called Dynamic Context Removal (DCR), which dynamically schedules the visibility of observed future in the learning procedure. It follows the human-like curriculum learning process, i.e., gradually removing the event context to increase the anticipation difficulty till satisfying the final anticipation target. Our learning scheme is plug-and-play and easy to integrate any reasoning model including transformer and LSTM, with advantages in both effectiveness and efficiency. In extensive experiments, the proposed method achieves state-of-the-art on four widely-used benchmarks. Our code and models are publicly released at https://github.com/AllenXuuu/DCR.