# Anticipating human actions by correlating past with the future with Jaccard similarity measures

arXiv: https://arxiv.org/abs/2105.12414 · HTML: https://mlsanity.com/paper/2105.12414/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 39 references that are in the map.
- First posted on arXiv 26 May 2021.

## Builds on (most cited first, 19 of 19)

- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [When will you do what? - Anticipating Temporal Occurrences of Activities](https://mlsanity.com/paper/1804.00892/) (2018)
- [What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attention](https://mlsanity.com/paper/1905.09035/) (2019)
- [RED: Reinforced Encoder-Decoder Networks for Action Anticipation](https://mlsanity.com/paper/1707.04818/) (2017)
- [Recurrent Neural Networks for Driver Activity Anticipation via Sensory-Fusion Architecture](https://mlsanity.com/paper/1509.05016/) (2015)
- [Predicting the Future: A Jointly Learnt Model for Action Anticipation](https://mlsanity.com/paper/1912.07148/) (2019)
- [Encouraging LSTMs to Anticipate Actions Very Early](https://mlsanity.com/paper/1703.07023/) (2017)
- [Multitask Learning to Improve Egocentric Action Recognition](https://mlsanity.com/paper/1909.06761/) (2019)
- [Predicting Human Activities Using Stochastic Grammar](https://mlsanity.com/paper/1708.00945/) (2017)
- [Action Anticipation with RBF Kernelized Feature Mapping RNN](https://mlsanity.com/paper/1911.07806/) (2019)
- [Forecasting future action sequences with attention: a new approach to weakly supervised action forecasting](https://mlsanity.com/paper/1912.04608/) (2019)
- [Adversarial Generative Grammars for Human Activity Prediction](https://mlsanity.com/paper/2008.04888/) (2020)
- [Action Anticipation By Predicting Future Dynamic Images](https://mlsanity.com/paper/1808.00141/) (2018)
- [Visual Forecasting by Imitating Dynamics in Natural Sequences](https://mlsanity.com/paper/1708.05827/) (2017)
- [Bregman Divergences for Infinite Dimensional Covariance Matrices](https://mlsanity.com/paper/1403.4334/) (2014)
- [TTPP: Temporal Transformer with Progressive Prediction for Efficient Action Anticipation](https://mlsanity.com/paper/2003.03530/) (2020)

## Built on by (most cited first, 30 of 36)

- [Future Transformer for Long-term Action Anticipation](https://mlsanity.com/paper/2205.14022/) (2022)
- [AdaPool: Exponential Adaptive Pooling for Information-Retaining Downsampling](https://mlsanity.com/paper/2111.00772/) (2021)
- [Interaction Region Visual Transformer for Egocentric Action Anticipation](https://mlsanity.com/paper/2211.14154/) (2022)
- [Can't make an Omelette without Breaking some Eggs: Plausible Action Anticipation using Large Video-Language Models](https://mlsanity.com/paper/2405.20305/) (2024)
- [Synchronization is All You Need: Exocentric-to-Egocentric Transfer for Temporal Action Segmentation with Unlabeled Synchronized Video Pairs](https://mlsanity.com/paper/2312.02638/) (2023)
- [Instance Tracking in 3D Scenes from Egocentric Videos](https://mlsanity.com/paper/2312.04117/) (2023)
- [Holistic Representation Learning for Multitask Trajectory Anomaly Detection](https://mlsanity.com/paper/2311.01851/) (2023)
- [Predicting the Next Action by Modeling the Abstract Goal](https://mlsanity.com/paper/2209.05044/) (2022)
- [The Wisdom of Crowds: Temporal Progressive Attention for Early Action Prediction](https://mlsanity.com/paper/2204.13340/) (2022)
- [A Survey on Deep Learning Techniques for Action Anticipation](https://mlsanity.com/paper/2309.17257/) (2023)
- [Human Action Anticipation: A Survey](https://mlsanity.com/paper/2410.14045/) (2024)
- [Adaptive Graph Convolutional Networks for Weakly Supervised Anomaly Detection in Videos](https://mlsanity.com/paper/2202.06503/) (2022)
- [Unified Recurrence Modeling for Video Action Anticipation](https://mlsanity.com/paper/2206.01009/) (2022)
- [Uncertainty-boosted Robust Video Activity Anticipation](https://mlsanity.com/paper/2404.18648/) (2024)
- [Learning to Discriminate Information for Online Action Detection: Analysis and Application](https://mlsanity.com/paper/2109.03393/) (2021)
- [Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction](https://mlsanity.com/paper/2107.05140/) (2021)
- [Towards Streaming Egocentric Action Anticipation](https://mlsanity.com/paper/2110.05386/) (2021)
- [FutureHuman3D: Forecasting Complex Long-Term 3D Human Behavior from Video Observations](https://mlsanity.com/paper/2211.14309/) (2022)
- [Streaming egocentric action anticipation: An evaluation scheme and approach](https://mlsanity.com/paper/2306.16682/) (2023)
- [Overlooked Poses Actually Make Sense: Distilling Privileged Knowledge for Human Motion Prediction](https://mlsanity.com/paper/2208.01302/) (2022)
- [VS-TransGRU: A Novel Transformer-GRU-based Framework Enhanced by Visual-Semantic Fusion for Egocentric Action Anticipation](https://mlsanity.com/paper/2307.03918/) (2023)
- [Knowledge-Guided Short-Context Action Anticipation in Human-Centric Videos](https://mlsanity.com/paper/2309.05943/) (2023)
- [Rich Action-semantic Consistent Knowledge for Early Action Prediction](https://mlsanity.com/paper/2201.09169/) (2022)
- [Leaping Into Memories: Space-Time Deep Feature Synthesis](https://mlsanity.com/paper/2303.09941/) (2023)
- [Multi-level and Multi-modal Action Anticipation](https://mlsanity.com/paper/2506.02382/) (2025)
- [MistExit: Learning to Exit for Early Mistake Detection in Procedural Videos](https://mlsanity.com/paper/2603.14252/) (2026)
- [EgoSpeed-Net: Forecasting Speed-Control in Driver Behavior from Egocentric Video Data](https://mlsanity.com/paper/2209.13459/) (2022)
- [GliTr: Glimpse Transformers with Spatiotemporal Consistency for Online Action Prediction](https://mlsanity.com/paper/2210.13605/) (2022)
- [Inferring Past Human Actions in Homes with Abductive Reasoning](https://mlsanity.com/paper/2210.13984/) (2022)
- [Inductive Attention for Video Action Anticipation](https://mlsanity.com/paper/2212.08830/) (2022)

## Abstract (arXiv)

We propose a framework for early action recognition and anticipation by correlating past features with the future using three novel similarity measures called Jaccard vector similarity, Jaccard cross-correlation and Jaccard Frobenius inner product over covariances. Using these combinations of novel losses and using our framework, we obtain state-of-the-art results for early action recognition in UCF101 and JHMDB datasets by obtaining 91.7 % and 83.5 % accuracy respectively for an observation percentage of 20. Similarly, we obtain state-of-the-art results for Epic-Kitchen55 and Breakfast datasets for action anticipation by obtaining 20.35 and 41.8 top-1 accuracy respectively.