# T2FPV: Dataset and Method for Correcting First-Person View Errors in Pedestrian Trajectory Prediction

arXiv: https://arxiv.org/abs/2209.11294 · HTML: https://mlsanity.com/paper/2209.11294/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 22 Sep 2022.

## Builds on (most cited first, 26 of 26)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks](https://mlsanity.com/paper/1803.10892/) (2018)
- [A Recurrent Latent Variable Model for Sequential Data](https://mlsanity.com/paper/1506.02216/) (2015)
- [Is Pseudo-Lidar needed for Monocular 3D Object detection?](https://mlsanity.com/paper/2108.06417/) (2021)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting](https://mlsanity.com/paper/2012.01526/) (2020)
- [The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections](https://mlsanity.com/paper/1911.07602/) (2019)
- [Core Challenges of Social Robot Navigation: A Survey](https://mlsanity.com/paper/2103.05668/) (2021)
- [The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction](https://mlsanity.com/paper/1912.06445/) (2019)
- [3D Object Detection from Images for Autonomous Driving: A Survey](https://mlsanity.com/paper/2202.02980/) (2022)
- [Human Trajectory Prediction via Neural Social Physics](https://mlsanity.com/paper/2207.10435/) (2022)
- [Probabilistic 3D Multi-Object Tracking for Autonomous Driving](https://mlsanity.com/paper/2001.05673/) (2020)
- [NAOMI: Non-Autoregressive Multiresolution Sequence Imputation](https://mlsanity.com/paper/1901.10946/) (2019)
- [Future Person Localization in First-Person Videos](https://mlsanity.com/paper/1711.11217/) (2017)
- [Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data](https://mlsanity.com/paper/2001.03093/) (2020)
- [View Vertically: A Hierarchical Network for Trajectory Prediction via Fourier Spectrums](https://mlsanity.com/paper/2110.07288/) (2021)
- [Egocentric Human Trajectory Forecasting with a Wearable Camera and Multi-Modal Fusion](https://mlsanity.com/paper/2111.00993/) (2021)
- [AC-VRNN: Attentive Conditional-VRNN for Multi-Future Trajectory Prediction](https://mlsanity.com/paper/2005.08307/) (2020)
- [MTP: Multi-Hypothesis Tracking and Prediction for Reduced Error Propagation](https://mlsanity.com/paper/2110.09481/) (2021)
- [SGNet: A Super-class Guided Network for Image Classification and Object Detection](https://mlsanity.com/paper/2104.12898/) (2021)
- [Egocentric Basketball Motion Planning from a Single First-Person Image](https://mlsanity.com/paper/1803.01413/) (2018)
- [An Approach to Deploy Interactive Robotic Simulators on the Web for HRI Experiments: Results in Social Robot Navigation](https://mlsanity.com/paper/2012.12336/) (2020)
- [Towards Robust Human Trajectory Prediction in Raw Videos](https://mlsanity.com/paper/2108.08259/) (2021)
- [DeepSocNav: Social Navigation by Imitating Human Behaviors](https://mlsanity.com/paper/2107.09170/) (2021)

## Built on by (most cited first, 5 of 5)

- [EgoTraj-Bench: Towards Robust Trajectory Prediction Under Ego-view Noisy Observations](https://mlsanity.com/paper/2510.00405/) (2025)
- [Sparse Pedestrian Character Learning for Trajectory Prediction](https://mlsanity.com/paper/2311.15512/) (2023)
- [Towards Human-Centered Construction Robotics: A Reinforcement Learning-Driven Companion Robot for Contextually Assisting Carpentry Workers](https://mlsanity.com/paper/2403.19060/) (2024)
- [Face versus Body Tracking for Human-Robot Interaction: An Egocentric Dataset](https://mlsanity.com/paper/2606.03694/) (2026)
- [Probabilistic Pedestrian Forecasts from a Handheld Phone: World-Frame Heat Maps, Visual-Inertial Height Drift, and Evaluation without Ground Truth](https://mlsanity.com/paper/2610.04736/) (2026)

## Abstract (arXiv)

Predicting pedestrian motion is essential for developing socially-aware robots that interact in a crowded environment. While the natural visual perspective for a social interaction setting is an egocentric view, the majority of existing work in trajectory prediction therein has been investigated purely in the top-down trajectory space. To support first-person view trajectory prediction research, we present T2FPV, a method for constructing high-fidelity first-person view (FPV) datasets given a real-world, top-down trajectory dataset; we showcase our approach on the ETH/UCY pedestrian dataset to generate the egocentric visual data of all interacting pedestrians, creating the T2FPV-ETH dataset. In this setting, FPV-specific errors arise due to imperfect detection and tracking, occlusions, and field-of-view (FOV) limitations of the camera. To address these errors, we propose CoFE, a module that further refines the imputation of missing data in an end-to-end manner with trajectory forecasting algorithms. Our method reduces the impact of such FPV errors on downstream prediction performance, decreasing displacement error by more than 10% on average. To facilitate research engagement, we release our T2FPV-ETH dataset and software tools.