# Towards Intelligent Transportation with Pedestrians and Vehicles In-the-Loop: A Surveillance Video-Assisted Federated Digital Twin Framework

arXiv: https://arxiv.org/abs/2503.04170 · HTML: https://mlsanity.com/paper/2503.04170/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 22 references that are in the map.
- First posted on arXiv 6 Mar 2025.

## Builds on (most cited first, 11 of 11)

- [Lightweight Frequency Masker for Cross-Domain Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2410.22135/) (2024)
- [GraCo: Granularity-Controllable Interactive Segmentation](https://mlsanity.com/paper/2405.00587/) (2024)
- [Towards Next Generation of Pedestrian and Connected Vehicle In-the-loop Research: A Digital Twin Co-Simulation Framework](https://mlsanity.com/paper/2212.05090/) (2022)
- [Networking Architecture and Key Supporting Technologies for Human Digital Twin in Personalized Healthcare: A Comprehensive Survey](https://mlsanity.com/paper/2301.03930/) (2023)
- [RISs and Sidelink Communications in Smart Cities: The Key to Seamless Localization and Sensing](https://mlsanity.com/paper/2301.03535/) (2023)
- [A Revolution of Personalized Healthcare: Enabling Human Digital Twin with Mobile AIGC](https://mlsanity.com/paper/2307.12115/) (2023)
- [Deep progressive reinforcement learning-based flexible resource scheduling framework for IRS and UAV-assisted MEC system](https://mlsanity.com/paper/2408.01248/) (2024)
- [Internet of Federated Digital Twins (IoFDT): Connecting Twins Beyond Borders for Society 5.0](https://mlsanity.com/paper/2312.06432/) (2023)
- [Dynamic Human Digital Twin Deployment at the Edge for Task Execution: A Two-Timescale Accuracy-Aware Online Optimization](https://mlsanity.com/paper/2401.16710/) (2024)
- [Ultrareliable and Low-Latency Communication Techniques for Tactile Internet Services](https://mlsanity.com/paper/1907.04474/) (2019)
- [Federated Learning for Digital Twin-Based Vehicular Networks: Architecture and Challenges](https://mlsanity.com/paper/2208.05558/) (2022)

## Built on by (most cited first, 1 of 1)

- [Network Digital Untwinning: Towards Backward Optimization of Digital Twins](https://mlsanity.com/paper/2605.00169/) (2026)

## Abstract (arXiv)

In intelligent transportation systems (ITSs), incorporating pedestrians and vehicles in-the-loop is crucial for developing realistic and safe traffic management solutions. However, there is falls short of simulating complex real-world ITS scenarios, primarily due to the lack of a digital twin implementation framework for characterizing interactions between pedestrians and vehicles at different locations in different traffic environments. In this article, we propose a surveillance video assisted federated digital twin (SV-FDT) framework to empower ITSs with pedestrians and vehicles in-the-loop. Specifically, SVFDT builds comprehensive pedestrian-vehicle interaction models by leveraging multi-source traffic surveillance videos. Its architecture consists of three layers: (i) the end layer, which collects traffic surveillance videos from multiple sources; (ii) the edge layer, responsible for semantic segmentation-based visual understanding, twin agent-based interaction modeling, and local digital twin system (LDTS) creation in local regions; and (iii) the cloud layer, which integrates LDTSs across different regions to construct a global DT model in realtime. We analyze key design requirements and challenges and present core guidelines for SVFDT's system implementation. A testbed evaluation demonstrates its effectiveness in optimizing traffic management. Comparisons with traditional terminal-server frameworks highlight SV-FDT's advantages in mirroring delays, recognition accuracy, and subjective evaluation. Finally, we identify some open challenges and discuss future research directions.