# Bidirectional Progressive Transformer for Interaction Intention Anticipation

arXiv: https://arxiv.org/abs/2405.05552 · HTML: https://mlsanity.com/paper/2405.05552/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 71 references that are in the map.
- First posted on arXiv 9 May 2024.

## Builds on (most cited first, 30 of 53)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [GMFlow: Learning Optical Flow via Global Matching](https://mlsanity.com/paper/2111.13680/) (2021)
- [Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction](https://mlsanity.com/paper/2002.11927/) (2020)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction](https://mlsanity.com/paper/2004.02025/) (2020)
- [Anticipative Video Transformer](https://mlsanity.com/paper/2106.02036/) (2021)
- [From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting](https://mlsanity.com/paper/2012.01526/) (2020)
- [Human-to-Robot Imitation in the Wild](https://mlsanity.com/paper/2207.09450/) (2022)
- [Hand-Object Contact Consistency Reasoning for Human Grasps Generation](https://mlsanity.com/paper/2104.03304/) (2021)
- [Structured World Models from Human Videos](https://mlsanity.com/paper/2308.10901/) (2023)
- [Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time](https://mlsanity.com/paper/2106.05266/) (2021)
- [Grounded Human-Object Interaction Hotspots from Video](https://mlsanity.com/paper/1812.04558/) (2018)
- [Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video](https://mlsanity.com/paper/2005.02190/) (2020)
- [UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer](https://mlsanity.com/paper/2211.09552/) (2022)
- [Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations](https://mlsanity.com/paper/2003.06085/) (2020)
- [Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos](https://mlsanity.com/paper/2204.01696/) (2022)

## Built on by (most cited first, 6 of 6)

- [GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Grounding](https://mlsanity.com/paper/2411.19626/) (2024)
- [EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views](https://mlsanity.com/paper/2405.13659/) (2024)
- [Grounding 3D Scene Affordance From Egocentric Interactions](https://mlsanity.com/paper/2409.19650/) (2024)
- [PEAR: Phrase-Based Hand-Object Interaction Anticipation](https://mlsanity.com/paper/2407.21510/) (2024)
- [Visual-Geometric Collaborative Guidance for Affordance Learning](https://mlsanity.com/paper/2410.11363/) (2024)
- [Unlocking 3D Affordance Segmentation with 2D Semantic Knowledge](https://mlsanity.com/paper/2510.08316/) (2025)

## Abstract (arXiv)

Interaction intention anticipation aims to jointly predict future hand trajectories and interaction hotspots. Existing research often treated trajectory forecasting and interaction hotspots prediction as separate tasks or solely considered the impact of trajectories on interaction hotspots, which led to the accumulation of prediction errors over time. However, a deeper inherent connection exists between hand trajectories and interaction hotspots, which allows for continuous mutual correction between them. Building upon this relationship, a novel Bidirectional prOgressive Transformer (BOT), which introduces a Bidirectional Progressive mechanism into the anticipation of interaction intention is established. Initially, BOT maximizes the utilization of spatial information from the last observation frame through the Spatial-Temporal Reconstruction Module, mitigating conflicts arising from changes of view in first-person videos. Subsequently, based on two independent prediction branches, a Bidirectional Progressive Enhancement Module is introduced to mutually improve the prediction of hand trajectories and interaction hotspots over time to minimize error accumulation. Finally, acknowledging the intrinsic randomness in human natural behavior, we employ a Trajectory Stochastic Unit and a C-VAE to introduce appropriate uncertainty to trajectories and interaction hotspots, respectively. Our method achieves state-of-the-art results on three benchmark datasets Epic-Kitchens-100, EGO4D, and EGTEA Gaze+, demonstrating superior in complex scenarios.