# TAMFormer: Multi-Modal Transformer with Learned Attention Mask for Early Intent Prediction

arXiv: https://arxiv.org/abs/2210.14714 · HTML: https://mlsanity.com/paper/2210.14714/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks](https://mlsanity.com/paper/1711.10305/) (2017)
- [A Short Note about Kinetics-600](https://mlsanity.com/paper/1808.01340/) (2018)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Beyond Short Snippets: Deep Networks for Video Classification](https://mlsanity.com/paper/1503.08909/) (2015)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognition](https://mlsanity.com/paper/2201.08383/) (2022)
- [STM: SpatioTemporal and Motion Encoding for Action Recognition](https://mlsanity.com/paper/1908.02486/) (2019)
- [AdaFrame: Adaptive Frame Selection for Fast Video Recognition](https://mlsanity.com/paper/1811.12432/) (2018)
- [Compressed Video Action Recognition](https://mlsanity.com/paper/1712.00636/) (2017)
- [What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attention](https://mlsanity.com/paper/1905.09035/) (2019)
- [VideoLSTM Convolves, Attends and Flows for Action Recognition](https://mlsanity.com/paper/1607.01794/) (2016)
- [Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors](https://mlsanity.com/paper/1505.04868/) (2015)
- [Knowledge Distillation for Action Anticipation via Label Smoothing](https://mlsanity.com/paper/2004.07711/) (2020)
- [SlowFast Rolling-Unrolling LSTMs for Action Anticipation in Egocentric Videos](https://mlsanity.com/paper/2109.00829/) (2021)
- [Am I Done? Predicting Action Progress in Videos](https://mlsanity.com/paper/1705.01781/) (2017)
- [Temporal Aggregate Representations for Long-Range Video Understanding](https://mlsanity.com/paper/2006.00830/) (2020)

## Built on by (most cited first, 9 of 9)

- [PIP-Net: Pedestrian Intention Prediction in the Wild](https://mlsanity.com/paper/2402.12810/) (2024)
- [Distilling Knowledge for Short-to-Long Term Trajectory Prediction](https://mlsanity.com/paper/2305.08553/) (2023)
- [Context-aware Multi-task Learning for Pedestrian Intent and Trajectory Prediction](https://mlsanity.com/paper/2407.17162/) (2024)
- [Multi-Context Fusion Transformer for Pedestrian Crossing Intention Prediction in Urban Environments](https://mlsanity.com/paper/2511.20011/) (2025)
- [MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding](https://mlsanity.com/paper/2507.12463/) (2025)
- [Early Action Recognition with Action Prototypes](https://mlsanity.com/paper/2312.06598/) (2023)
- [MUSCLE-NET: Predicted-Multiscale-Aware Network for Pedestrian Trajectory Forecasting](https://mlsanity.com/paper/2606.00471/) (2026)
- [Adaptive Cross-Modal Fusion with Sparse Attention for Pedestrian Crossing Intention Prediction](https://mlsanity.com/paper/2607.12293/) (2026)
- [Unified Vision-Centric Pedestrian Crossing Action Prediction via Adaptive Patch Projection and Proactive Spatial Rectification](https://mlsanity.com/paper/2609.07420/) (2026)

## Abstract (arXiv)

Human intention prediction is a growing area of research where an activity in a video has to be anticipated by a vision-based system. To this end, the model creates a representation of the past, and subsequently, it produces future hypotheses about upcoming scenarios. In this work, we focus on pedestrians' early intention prediction in which, from a current observation of an urban scene, the model predicts the future activity of pedestrians that approach the street. Our method is based on a multi-modal transformer that encodes past observations and produces multiple predictions at different anticipation times. Moreover, we propose to learn the attention masks of our transformer-based model (Temporal Adaptive Mask Transformer) in order to weigh differently present and past temporal dependencies. We investigate our method on several public benchmarks for early intention prediction, improving the prediction performances at different anticipation times compared to the previous works.