# Finding Islands of Predictability in Action Forecasting

arXiv: https://arxiv.org/abs/2210.07354 · HTML: https://mlsanity.com/paper/2210.07354/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 18 of 18)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [Autonomous Agents Modelling Other Agents: A Comprehensive Survey and Open Problems](https://mlsanity.com/paper/1709.08071/) (2017)
- [When will you do what? - Anticipating Temporal Occurrences of Activities](https://mlsanity.com/paper/1804.00892/) (2018)
- [D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation](https://mlsanity.com/paper/1901.02598/) (2019)
- [Lightweight Probabilistic Deep Networks](https://mlsanity.com/paper/1805.11327/) (2018)
- [Learning the Predictability of the Future](https://mlsanity.com/paper/2101.01600/) (2021)
- [Zero-Shot Anticipation for Instructional Activities](https://mlsanity.com/paper/1812.02501/) (2018)
- [Long-Term Anticipation of Activities with Cycle Consistency](https://mlsanity.com/paper/2009.01142/) (2020)
- [Uncertainty-Aware Anticipation of Activities](https://mlsanity.com/paper/1908.09540/) (2019)
- [A Variational Auto-Encoder Model for Stochastic Point Processes](https://mlsanity.com/paper/1904.03273/) (2019)
- [Forecasting future action sequences with attention: a new approach to weakly supervised action forecasting](https://mlsanity.com/paper/1912.04608/) (2019)
- [Adversarial Generative Grammars for Human Activity Prediction](https://mlsanity.com/paper/2008.04888/) (2020)
- [Generative Hybrid Representations for Activity Forecasting with No-Regret Learning](https://mlsanity.com/paper/1904.06250/) (2019)
- [Forecasting Future Action Sequences with Neural Memory Networks](https://mlsanity.com/paper/1909.09278/) (2019)
- [A hierarchical loss and its problems when classifying non-hierarchically](https://mlsanity.com/paper/1709.01062/) (2017)
- [Learning to Abstract and Predict Human Actions](https://mlsanity.com/paper/2008.09234/) (2020)

## Abstract (arXiv)

We address dense action forecasting: the problem of predicting future action sequence over long durations based on partial observation. Our key insight is that future action sequences are more accurately modeled with variable, rather than one, levels of abstraction, and that the optimal level of abstraction can be dynamically selected during the prediction process. Our experiments show that most parts of future action sequences can be predicted confidently in fine detail only in small segments of future frames, which are effectively ``islands'' of high model prediction confidence in a ``sea'' of uncertainty. We propose a combination Bayesian neural network and hierarchical convolutional segmentation model to both accurately predict future actions and optimally select abstraction levels. We evaluate this approach on standard datasets against existing state-of-the-art systems and demonstrate that our ``islands of predictability'' approach maintains fine-grained action predictions while also making accurate abstract predictions where systems were previously unable to do so, and thus results in substantial, monotonic increases in accuracy.