# JRDB-Traj: A Dataset and Benchmark for Trajectory Forecasting in Crowds

arXiv: https://arxiv.org/abs/2311.02736 · HTML: https://mlsanity.com/paper/2311.02736/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 14 references that are in the map.
- First posted on arXiv 5 Nov 2023.

## Builds on (most cited first, 10 of 10)

- [Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking](https://mlsanity.com/paper/1609.01775/) (2016)
- [Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning](https://mlsanity.com/paper/1809.08835/) (2018)
- [Human Trajectory Forecasting in Crowds: A Deep Learning Perspective](https://mlsanity.com/paper/2007.03639/) (2020)
- [SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking](https://mlsanity.com/paper/2111.09621/) (2021)
- [JRDB-Act: A Large-scale Dataset for Spatio-temporal Action, Social Group and Activity Detection](https://mlsanity.com/paper/2106.08827/) (2021)
- [PORCA: Modeling and Planning for Autonomous Driving among Many Pedestrians](https://mlsanity.com/paper/1805.11833/) (2018)
- [Are socially-aware trajectory prediction models really socially-aware?](https://mlsanity.com/paper/2108.10879/) (2021)
- [JRDB-Pose: A Large-scale Dataset for Multi-Person Pose Estimation and Tracking](https://mlsanity.com/paper/2210.11940/) (2022)
- [How Trustworthy are Performance Evaluations for Basic Vision Tasks?](https://mlsanity.com/paper/2008.03533/) (2020)
- [Accurate and Real-time 3D Pedestrian Detection Using an Efficient Attentive Pillar Network](https://mlsanity.com/paper/2112.15458/) (2021)

## Built on by (most cited first, 4 of 4)

- [Multi-Agent Inverse Reinforcement Learning in Real World Unstructured Pedestrian Crowds](https://mlsanity.com/paper/2405.16439/) (2024)
- [Diverse Yet Consistent: Context-Guided Diffusion with Energy-Based Joint Refinement for Multi-Agent Motion Prediction](https://mlsanity.com/paper/2605.22017/) (2026)
- [EgoNeMo: Transferable Map of Pedestrian Dynamics via Egocentric LiDAR Scan](https://mlsanity.com/paper/2609.06195/) (2026)
- [JRDB-AVR: An Active Visual Reasoning Benchmark for Embodied Agents in Real-World Environments](https://mlsanity.com/paper/2609.35032/) (2026)

## Abstract (arXiv)

Predicting future trajectories is critical in autonomous navigation, especially in preventing accidents involving humans, where a predictive agent's ability to anticipate in advance is of utmost importance. Trajectory forecasting models, employed in fields such as robotics, autonomous vehicles, and navigation, face challenges in real-world scenarios, often due to the isolation of model components. To address this, we introduce a novel dataset for end-to-end trajectory forecasting, facilitating the evaluation of models in scenarios involving less-than-ideal preceding modules such as tracking. This dataset, an extension of the JRDB dataset, provides comprehensive data, including the locations of all agents, scene images, and point clouds, all from the robot's perspective. The objective is to predict the future positions of agents relative to the robot using raw sensory input data. It bridges the gap between isolated models and practical applications, promoting a deeper understanding of navigation dynamics. Additionally, we introduce a novel metric for assessing trajectory forecasting models in real-world scenarios where ground-truth identities are inaccessible, addressing issues related to undetected or over-detected agents. Researchers are encouraged to use our benchmark for model evaluation and benchmarking.