# VisionTraj: A Noise-Robust Trajectory Recovery Framework based on Large-scale Camera Network

arXiv: https://arxiv.org/abs/2312.06428 · HTML: https://mlsanity.com/paper/2312.06428/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 53 references that are in the map.
- First posted on arXiv 11 Dec 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections](https://mlsanity.com/paper/1911.07602/) (2019)
- [Unsupervised Path Representation Learning with Curriculum Negative Sampling](https://mlsanity.com/paper/2106.09373/) (2021)
- [A survey of advances in vision-based vehicle re-identification](https://mlsanity.com/paper/1905.13258/) (2019)
- [Simulating Content Consistent Vehicle Datasets with Attribute Descent](https://mlsanity.com/paper/1912.08855/) (2019)
- [Jointly Contrastive Representation Learning on Road Network and Trajectory](https://mlsanity.com/paper/2209.06389/) (2022)
- [RNTrajRec: Road Network Enhanced Trajectory Recovery with Spatial-Temporal Transformer](https://mlsanity.com/paper/2211.13234/) (2022)
- [City-Scale Multi-Camera Vehicle Tracking Guided by Crossroad Zones](https://mlsanity.com/paper/2105.06623/) (2021)
- [Long-Short Term Spatiotemporal Tensor Prediction for Passenger Flow Profile](https://mlsanity.com/paper/2004.11022/) (2020)
- [Dynamic Causal Graph Convolutional Network for Traffic Prediction](https://mlsanity.com/paper/2306.07019/) (2023)
- [Correlated Time Series Self-Supervised Representation Learning via Spatiotemporal Bootstrapping](https://mlsanity.com/paper/2306.06994/) (2023)
- [Trajectory Forecasting from Detection with Uncertainty-Aware Motion Encoding](https://mlsanity.com/paper/2202.01478/) (2022)
- [Adaptive Hierarchical SpatioTemporal Network for Traffic Forecasting](https://mlsanity.com/paper/2306.09386/) (2023)
- [Tensor Dirichlet Process Multinomial Mixture Model for Passenger Trajectory Clustering](https://mlsanity.com/paper/2306.13794/) (2023)
- [A Critical Perceptual Pre-trained Model for Complex Trajectory Recovery](https://mlsanity.com/paper/2311.02631/) (2023)

## Built on by (most cited first, 3 of 3)

- [Trajectory Data Management and Mining: A Survey from Deep Learning to the LLM Era](https://mlsanity.com/paper/2403.14151/) (2024)
- [Skin Cancer Detection utilizing Deep Learning: Classification of Skin Lesion Images using a Vision Transformer](https://mlsanity.com/paper/2407.18554/) (2024)
- [Exploring Genre and Success Classification through Song Lyrics using DistilBERT: A Fun NLP Venture](https://mlsanity.com/paper/2407.21068/) (2024)

## Abstract (arXiv)

Trajectory recovery based on the snapshots from the city-wide multi-camera network facilitates urban mobility sensing and driveway optimization. The state-of-the-art solutions devoted to such a vision-based scheme typically incorporate predefined rules or unsupervised iterative feedback, struggling with multi-fold challenges such as lack of open-source datasets for training the whole pipeline, and the vulnerability to the noises from visual inputs. In response to the dilemma, this paper proposes VisionTraj, the first learning-based model that reconstructs vehicle trajectories from snapshots recorded by road network cameras. Coupled with it, we elaborate on two rational vision-trajectory datasets, which produce extensive trajectory data along with corresponding visual snapshots, enabling supervised vision-trajectory interplay extraction. Following the data creation, based on the results from the off-the-shelf multi-modal vehicle clustering, we first re-formulate the trajectory recovery problem as a generative task and introduce the canonical Transformer as the autoregressive backbone. Then, to identify clustering noises (e.g., false positives) with the bound on the snapshots' spatiotemporal dependencies, a GCN-based soft-denoising module is conducted based on the fine- and coarse-grained Re-ID clusters. Additionally, we harness strong semantic information extracted from the tracklet to provide detailed insights into the vehicle's entry and exit actions during trajectory recovery. The denoising and tracklet components can also act as plug-and-play modules to boost baselines. Experimental results on the two hand-crafted datasets show that the proposed VisionTraj achieves a maximum +11.5% improvement against the sub-best model.