# CitySim: A Drone-Based Vehicle Trajectory Dataset for Safety Oriented Research and Digital Twins

arXiv: https://arxiv.org/abs/2208.11036 · HTML: https://mlsanity.com/paper/2208.11036/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 58 references that are in the map.
- First posted on arXiv 23 Aug 2022.

## Builds on (most cited first, 17 of 17)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems](https://mlsanity.com/paper/1810.05642/) (2018)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [Feature Selective Anchor-Free Module for Single-Shot Object Detection](https://mlsanity.com/paper/1903.00621/) (2019)
- [The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections](https://mlsanity.com/paper/1911.07602/) (2019)
- [Active Learning for Deep Object Detection](https://mlsanity.com/paper/1809.09875/) (2018)
- [Driving Behavior Modeling using Naturalistic Human Driving Data with Inverse Reinforcement Learning](https://mlsanity.com/paper/2010.03118/) (2020)
- [Modeling Human Driving Behavior through Generative Adversarial Imitation Learning](https://mlsanity.com/paper/2006.06412/) (2020)
- [Consistency-based Active Learning for Object Detection](https://mlsanity.com/paper/2103.10374/) (2021)
- [Image Matching Using SIFT, SURF, BRIEF and ORB: Performance Comparison for Distorted Images](https://mlsanity.com/paper/1710.02726/) (2017)
- [Digital Twin-Assisted Cooperative Driving at Non-Signalized Intersections](https://mlsanity.com/paper/2105.01357/) (2021)

## Built on by (most cited first, 30 of 43)

- [DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Models](https://mlsanity.com/paper/2309.16292/) (2023)
- [On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving](https://mlsanity.com/paper/2311.05332/) (2023)
- [TrafficSafetyGPT: Tuning a Pre-trained Large Language Model to a Domain-Specific Expert in Transportation Safety](https://mlsanity.com/paper/2307.15311/) (2023)
- [I-24 MOTION: An instrument for freeway traffic science](https://mlsanity.com/paper/2301.11198/) (2023)
- [Laplacian Convolutional Representation for Traffic Time Series Imputation](https://mlsanity.com/paper/2212.01529/) (2022)
- [Multiview Aerial Visual Recognition (MAVREC): Can Multi-view Improve Aerial Visual Perception?](https://mlsanity.com/paper/2312.04548/) (2023)
- [Advanced computer vision for extracting georeferenced vehicle trajectories from drone imagery](https://mlsanity.com/paper/2411.02136/) (2024)
- [Car-Following Models: A Multidisciplinary Review](https://mlsanity.com/paper/2304.07143/) (2023)
- [CAROM Air -- Vehicle Localization and Traffic Scene Reconstruction from Aerial Videos](https://mlsanity.com/paper/2306.00075/) (2023)
- [IntersectionZoo: Eco-driving for Benchmarking Multi-Agent Contextual Reinforcement Learning](https://mlsanity.com/paper/2410.15221/) (2024)
- [Highly Accurate and Diverse Traffic Data: The DeepScenario Open 3D Dataset](https://mlsanity.com/paper/2504.17371/) (2025)
- [Trajectory Prediction for Vehicle Conflict Identification at Intersections Using Sequence-to-Sequence Recurrent Neural Networks](https://mlsanity.com/paper/2210.08009/) (2022)
- [Bringing Diversity to Autonomous Vehicles: An Interpretable Multi-vehicle Decision-making and Planning Framework](https://mlsanity.com/paper/2302.06803/) (2023)
- [A Novel Temporal Multi-Gate Mixture-of-Experts Approach for Vehicle Trajectory and Driving Intention Prediction](https://mlsanity.com/paper/2308.00533/) (2023)
- [Advances and Applications of Computer Vision Techniques in Vehicle Trajectory Generation and Surrogate Traffic Safety Indicators](https://mlsanity.com/paper/2303.15231/) (2023)
- [Driving-RAG: Driving Scenarios Embedding, Search, and RAG Applications](https://mlsanity.com/paper/2504.04419/) (2025)
- [DrivIng: A Large-Scale Multimodal Driving Dataset with Full Digital Twin Integration](https://mlsanity.com/paper/2601.15260/) (2026)
- [A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous driving](https://mlsanity.com/paper/2607.07103/) (2026)
- [Interaction-Aware Personalized Vehicle Trajectory Prediction Using Temporal Graph Neural Networks](https://mlsanity.com/paper/2308.07439/) (2023)
- [Digital Twin-based Driver Risk-Aware Intelligent Mobility Analytics for Urban Transportation Management](https://mlsanity.com/paper/2407.15025/) (2024)
- [Mitigating Metropolitan Carbon Emissions with Dynamic Eco-driving at Scale](https://mlsanity.com/paper/2408.05609/) (2024)
- [ATARS: An Aerial Traffic Atomic Activity Recognition and Temporal Segmentation Dataset](https://mlsanity.com/paper/2503.18553/) (2025)
- [Game-Theoretic Modeling of Vehicle Unprotected Left Turns Considering Drivers' Bounded Rationality](https://mlsanity.com/paper/2507.03002/) (2025)
- [UAV-Based Intelligent Traffic Surveillance System: Real-Time Vehicle Detection, Classification, Tracking, and Behavioral Analysis](https://mlsanity.com/paper/2509.04624/) (2025)
- [Spatial-Temporal Deep Embedding for Vehicle Trajectory Reconstruction from High-Angle Video](https://mlsanity.com/paper/2209.08417/) (2022)
- [Incorporating Nonlocal Traffic Flow Model in Physics-informed Neural Networks](https://mlsanity.com/paper/2308.11818/) (2023)
- [TrTr: A Versatile Pre-Trained Large Traffic Model based on Transformer for Capturing Trajectory Diversity in Vehicle Population](https://mlsanity.com/paper/2309.12677/) (2023)
- [Data-Driven Semi-Supervised Machine Learning with Safety Indicators for Abnormal Driving Behavior Detection](https://mlsanity.com/paper/2312.04610/) (2023)
- [Real-Time Vehicle Detection and Urban Traffic Behavior Analysis Based on UAV Traffic Videos on Mobile Devices](https://mlsanity.com/paper/2402.16246/) (2024)
- [Interpretable Interaction Modeling for Trajectory Prediction via Agent Selection and Physical Coefficient](https://mlsanity.com/paper/2405.13152/) (2024)

## Abstract (arXiv)

The development of safety-oriented research and applications requires fine-grain vehicle trajectories that not only have high accuracy, but also capture substantial safety-critical events. However, it would be challenging to satisfy both these requirements using the available vehicle trajectory datasets do not have the capacity to satisfy both.This paper introduces the CitySim dataset that has the core objective of facilitating safety-oriented research and applications. CitySim has vehicle trajectories extracted from 1140 minutes of drone videos recorded at 12 locations. It covers a variety of road geometries including freeway basic segments, signalized intersections, stop-controlled intersections, and control-free intersections. CitySim was generated through a five-step procedure that ensured trajectory accuracy. The five-step procedure included video stabilization, object filtering, multi-video stitching, object detection and tracking, and enhanced error filtering. Furthermore, CitySim provides the rotated bounding box information of a vehicle, which was demonstrated to improve safety evaluations. Compared with other video-based critical events, including cut-in, merge, and diverge events, which were validated by distributions of both minimum time-to-collision and minimum post-encroachment time. In addition, CitySim had the capability to facilitate digital-twin-related research by providing relevant assets, such as the recording locations' three-dimensional base maps and signal timings.