# Unified Human Localization and Trajectory Prediction with Monocular Vision

arXiv: https://arxiv.org/abs/2503.03535 · HTML: https://mlsanity.com/paper/2503.03535/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 41 references that are in the map.
- First posted on arXiv 5 Mar 2025.

## Builds on (most cited first, 30 of 34)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [Categorical Depth Distribution Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.01100/) (2021)
- [SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints](https://mlsanity.com/paper/1806.01482/) (2018)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras](https://mlsanity.com/paper/2104.10490/) (2021)
- [Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net](https://mlsanity.com/paper/2012.12395/) (2020)
- [PnPNet: End-to-End Perception and Prediction with Tracking in the Loop](https://mlsanity.com/paper/2005.14711/) (2020)
- [Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification](https://mlsanity.com/paper/2302.11813/) (2023)
- [Latent Variable Sequential Set Transformers For Joint Multi-Agent Motion Prediction](https://mlsanity.com/paper/2104.00563/) (2021)
- [Human Trajectory Forecasting in Crowds: A Deep Learning Perspective](https://mlsanity.com/paper/2007.03639/) (2020)
- [ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries](https://mlsanity.com/paper/2208.01582/) (2022)
- [Kinematic 3D Object Detection in Monocular Video](https://mlsanity.com/paper/2007.09548/) (2020)
- [MonoRUn: Monocular 3D Object Detection by Reconstruction and Uncertainty Propagation](https://mlsanity.com/paper/2103.12605/) (2021)
- [Social NCE: Contrastive Learning of Socially-aware Motion Representations](https://mlsanity.com/paper/2012.11717/) (2020)
- [OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association](https://mlsanity.com/paper/2103.02440/) (2021)
- [UniTraj: A Unified Framework for Scalable Vehicle Trajectory Prediction](https://mlsanity.com/paper/2403.15098/) (2024)
- [Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction](https://mlsanity.com/paper/1710.09483/) (2017)
- [Multimodal Deep Generative Models for Trajectory Prediction: A Conditional Variational Autoencoder Approach](https://mlsanity.com/paper/2008.03880/) (2020)
- [MonoLoco: Monocular 3D Pedestrian Localization and Uncertainty Estimation](https://mlsanity.com/paper/1906.06059/) (2019)
- [Social-Transmotion: Promptable Human Trajectory Prediction](https://mlsanity.com/paper/2312.16168/) (2023)
- [Review of Pedestrian Trajectory Prediction Methods: Comparing Deep Learning and Knowledge-based Approaches](https://mlsanity.com/paper/2111.06740/) (2021)
- [Towards Motion Forecasting with Real-World Perception Inputs: Are End-to-End Approaches Competitive?](https://mlsanity.com/paper/2306.09281/) (2023)
- [MonoPGC: Monocular 3D Object Detection with Pixel Geometry Contexts](https://mlsanity.com/paper/2302.10549/) (2023)
- [Multi-Transmotion: Pre-trained Model for Human Motion Prediction](https://mlsanity.com/paper/2411.02673/) (2024)
- [Optimizing Diffusion Models for Joint Trajectory Prediction and Controllable Generation](https://mlsanity.com/paper/2408.00766/) (2024)

## Built on by (most cited first, 5 of 5)

- [Social-Pose: Enhancing Trajectory Prediction with Human Body Pose](https://mlsanity.com/paper/2507.22742/) (2025)
- [OmniTraj: Pre-Training on Heterogeneous Data for Adaptive and Zero-Shot Human Trajectory Prediction](https://mlsanity.com/paper/2507.23657/) (2025)
- [Social-Mamba: Socially-Aware Trajectory Forecasting with State-Space Models](https://mlsanity.com/paper/2605.15424/) (2026)
- [Deformable Gaussian Occupancy: Decoupling Rigid and Nonrigid Motion with Factorized Distillation](https://mlsanity.com/paper/2605.28587/) (2026)
- [Probabilistic Pedestrian Forecasts from a Handheld Phone: World-Frame Heat Maps, Visual-Inertial Height Drift, and Evaluation without Ground Truth](https://mlsanity.com/paper/2610.04736/) (2026)

## Abstract (arXiv)

Conventional human trajectory prediction models rely on clean curated data, requiring specialized equipment or manual labeling, which is often impractical for robotic applications. The existing predictors tend to overfit to clean observation affecting their robustness when used with noisy inputs. In this work, we propose MonoTransmotion (MT), a Transformer-based framework that uses only a monocular camera to jointly solve localization and prediction tasks. Our framework has two main modules: Bird's Eye View (BEV) localization and trajectory prediction. The BEV localization module estimates the position of a person using 2D human poses, enhanced by a novel directional loss for smoother sequential localizations. The trajectory prediction module predicts future motion from these estimates. We show that by jointly training both tasks with our unified framework, our method is more robust in real-world scenarios made of noisy inputs. We validate our MT network on both curated and non-curated datasets. On the curated dataset, MT achieves around 12% improvement over baseline models on BEV localization and trajectory prediction. On real-world non-curated dataset, experimental results indicate that MT maintains similar performance levels, highlighting its robustness and generalization capability. The code is available at https://github.com/vita-epfl/MonoTransmotion.