# MX-LSTM: mixing tracklets and vislets to jointly forecast trajectories and head poses

arXiv: https://arxiv.org/abs/1805.00652 · HTML: https://mlsanity.com/paper/1805.00652/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2018.

## Built on by (most cited first, 17 of 17)

- [DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets](https://mlsanity.com/paper/2108.09640/) (2021)
- [Trajectory Prediction with Latent Belief Energy-Based Model](https://mlsanity.com/paper/2104.03086/) (2021)
- [How many Observations are Enough? Knowledge Distillation for Trajectory Forecasting](https://mlsanity.com/paper/2203.04781/) (2022)
- [Review of Pedestrian Trajectory Prediction Methods: Comparing Deep Learning and Knowledge-based Approaches](https://mlsanity.com/paper/2111.06740/) (2021)
- [PedFormer: Pedestrian Behavior Prediction via Cross-Modal Attention Modulation and Gated Multitask Learning](https://mlsanity.com/paper/2210.07886/) (2022)
- [Collaborative Uncertainty in Multi-Agent Trajectory Forecasting](https://mlsanity.com/paper/2110.13947/) (2021)
- [Trajectory Prediction for Autonomous Driving: Progress, Limitations, and Future Directions](https://mlsanity.com/paper/2503.03262/) (2025)
- [Collaborative Uncertainty Benefits Multi-Agent Multi-Modal Trajectory Forecasting](https://mlsanity.com/paper/2207.05195/) (2022)
- [Safety-compliant Generative Adversarial Networks for Human Trajectory Forecasting](https://mlsanity.com/paper/2209.12243/) (2022)
- [Under the Hood of Transformer Networks for Trajectory Forecasting](https://mlsanity.com/paper/2203.11878/) (2022)
- [Towards Robust Human Trajectory Prediction in Raw Videos](https://mlsanity.com/paper/2108.08259/) (2021)
- [IA-LSTM: Interaction-Aware LSTM for Pedestrian Trajectory Prediction](https://mlsanity.com/paper/2311.15193/) (2023)
- [Handling Missing Observations with an RNN-based Prediction-Update Cycle](https://mlsanity.com/paper/2103.11747/) (2021)
- [3D Head-Position Prediction in First-Person View by Considering Head Pose for Human-Robot Eye Contact](https://mlsanity.com/paper/2103.06417/) (2021)
- [Eye Gaze-Informed and Context-Aware Pedestrian Trajectory Prediction in Shared Spaces with Automated Shuttles: A Virtual Reality Study](https://mlsanity.com/paper/2603.19812/) (2026)
- [Human motion trajectory prediction using the Social Force Model for real-time and low computational cost applications](https://mlsanity.com/paper/2311.10582/) (2023)
- [EMT: A Visual Multi-Task Benchmark Dataset for Autonomous Driving](https://mlsanity.com/paper/2502.19260/) (2025)

## Abstract (arXiv)

Recent approaches on trajectory forecasting use tracklets to predict the future positions of pedestrians exploiting Long Short Term Memory (LSTM) architectures. This paper shows that adding vislets, that is, short sequences of head pose estimations, allows to increase significantly the trajectory forecasting performance. We then propose to use vislets in a novel framework called MX-LSTM, capturing the interplay between tracklets and vislets thanks to a joint unconstrained optimization of full covariance matrices during the LSTM backpropagation. At the same time, MX-LSTM predicts the future head poses, increasing the standard capabilities of the long-term trajectory forecasting approaches. With standard head pose estimators and an attentional-based social pooling, MX-LSTM scores the new trajectory forecasting state-of-the-art in all the considered datasets (Zara01, Zara02, UCY, and TownCentre) with a dramatic margin when the pedestrians slow down, a case where most of the forecasting approaches struggle to provide an accurate solution.