# Towards a large-scale fused and labeled dataset of human pose while interacting with robots in shared urban areas

arXiv: https://arxiv.org/abs/2402.10077 · HTML: https://mlsanity.com/paper/2402.10077/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 28 Jan 2024.

## Builds on (most cited first, 15 of 15)

- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors](https://mlsanity.com/paper/2207.02696/) (2022)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [MOT16: A Benchmark for Multi-Object Tracking](https://mlsanity.com/paper/1603.00831/) (2016)
- [Long-term Human Motion Prediction with Scene Context](https://mlsanity.com/paper/2007.03672/) (2020)
- [Multi-Person 3D Motion Prediction with Multi-Range Transformers](https://mlsanity.com/paper/2111.12073/) (2021)
- [TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild](https://mlsanity.com/paper/2104.04029/) (2021)
- [Learning 3D Human Pose Estimation from Dozens of Datasets using a Geometry-Aware Autoencoder to Bridge Between Skeleton Formats](https://mlsanity.com/paper/2212.14474/) (2022)
- [Multi-modal 3D Human Pose Estimation with 2D Weak Supervision in Autonomous Driving](https://mlsanity.com/paper/2112.12141/) (2021)
- [Intention Recognition of Pedestrians and Cyclists by 2D Pose Estimation](https://mlsanity.com/paper/1910.03858/) (2019)
- [Forecasting Characteristic 3D Poses of Human Actions](https://mlsanity.com/paper/2011.15079/) (2020)
- [A context-aware pedestrian trajectory prediction framework for automated vehicles](https://mlsanity.com/paper/2104.08123/) (2021)
- [Dyadic Human Motion Prediction](https://mlsanity.com/paper/2112.00396/) (2021)
- [Weakly Supervised Multi-Modal 3D Human Body Pose Estimation for Autonomous Driving](https://mlsanity.com/paper/2307.14889/) (2023)
- [Virtual Immersive Reality for Stated Preference Travel Behaviour Experiments: A Case Study of Autonomous Vehicles on Urban Roads](https://mlsanity.com/paper/1802.06180/) (2018)

## Abstract (arXiv)

Over the last decade, Autonomous Delivery Robots (ADRs) have transformed conventional delivery methods, responding to the growing e-commerce demand. However, the readiness of ADRs to navigate safely among pedestrians in shared urban areas remains an open question. We contend that there are crucial research gaps in understanding their interactions with pedestrians in such environments. Human Pose Estimation is a vital stepping stone for various downstream applications, including pose prediction and socially aware robot path-planning. Yet, the absence of an enriched and pose-labeled dataset capturing human-robot interactions in shared urban areas hinders this objective. In this paper, we bridge this gap by repurposing, fusing, and labeling two datasets, MOT17 and NCLT, focused on pedestrian tracking and Simultaneous Localization and Mapping (SLAM), respectively. The resulting unique dataset represents thousands of real-world indoor and outdoor human-robot interaction scenarios. Leveraging YOLOv7, we obtained human pose visual and numeric outputs and provided ground truth poses using manual annotation. To overcome the distance bias present in the traditional MPJPE metric, this study introduces a novel human pose estimation error metric called Mean Scaled Joint Error (MSJE) by incorporating bounding box dimensions into it. Findings demonstrate that YOLOv7 effectively estimates human pose in both datasets. However, it exhibits weaker performance in specific scenarios, like indoor, crowded scenes with a focused light source, where both MPJPE and MSJE are recorded as 10.89 and 25.3, respectively. In contrast, YOLOv7 performs better in single-person estimation (NCLT seq 2) and outdoor scenarios (MOT17 seq1), achieving MSJE values of 5.29 and 3.38, respectively.