# JRDB: A Dataset and Benchmark of Egocentric Robot Visual Perception of Humans in Built Environments

arXiv: https://arxiv.org/abs/1910.11792 · HTML: https://mlsanity.com/paper/1910.11792/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2019.

## Built on by (most cited first, 12 of 12)

- [Principles and Guidelines for Evaluating Social Robot Navigation Algorithms](https://mlsanity.com/paper/2306.16740/) (2023)
- [Towards Robust Robot 3D Perception in Urban Environments: The UT Campus Object Dataset](https://mlsanity.com/paper/2309.13549/) (2023)
- [Know Your Surroundings: Panoramic Multi-Object Tracking by Multimodality Collaboration](https://mlsanity.com/paper/2105.14683/) (2021)
- [2D vs. 3D LiDAR-based Person Detection on Mobile Robots](https://mlsanity.com/paper/2106.11239/) (2021)
- [Cross-Modal Analysis of Human Detection for Robotics: An Industrial Case Study](https://mlsanity.com/paper/2108.01495/) (2021)
- [A Neuro-Symbolic Approach for Enhanced Human Motion Prediction](https://mlsanity.com/paper/2304.11740/) (2023)
- [Person-MinkUNet: 3D Person Detection with LiDAR Point Cloud](https://mlsanity.com/paper/2107.06780/) (2021)
- [Predicting Human Perceptions of Robot Performance During Navigation Tasks](https://mlsanity.com/paper/2310.11590/) (2023)
- [Valeo Near-Field: a novel dataset for pedestrian intent detection](https://mlsanity.com/paper/2510.15673/) (2025)
- [Vision-Language Models for Deployable Social Robot Navigation: Bridging Semantic Reasoning and Low-Level Control](https://mlsanity.com/paper/2606.28760/) (2026)
- [Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence](https://mlsanity.com/paper/2609.29669/) (2026)
- [STARS: From Spatiotemporal Dynamics to Social Representations in Human-Robot Interaction](https://mlsanity.com/paper/2609.40245/) (2026)

## Abstract (arXiv)

We present JRDB, a novel egocentric dataset collected from our social mobile manipulator JackRabbot. The dataset includes 64 minutes of annotated multimodal sensor data including stereo cylindrical 360^circ RGB video at 15 fps, 3D point clouds from two Velodyne 16 Lidars, line 3D point clouds from two Sick Lidars, audio signal, RGB-D video at 30 fps, 360^circ spherical image from a fisheye camera and encoder values from the robot's wheels. Our dataset incorporates data from traditionally underrepresented scenes such as indoor environments and pedestrian areas, all from the ego-perspective of the robot, both stationary and navigating. The dataset has been annotated with over 2.3 million bounding boxes spread over 5 individual cameras and 1.8 million associated 3D cuboids around all people in the scenes totaling over 3500 time consistent trajectories. Together with our dataset and the annotations, we launch a benchmark and metrics for 2D and 3D person detection and tracking. With this dataset, which we plan on extending with further types of annotation in the future, we hope to provide a new source of data and a test-bench for research in the areas of egocentric robot vision, autonomous navigation, and all perceptual tasks around social robotics in human environments.