# Novelty Detection and Analysis of Traffic Scenario Infrastructures in the Latent Space of a Vision Transformer-Based Triplet Autoencoder

arXiv: https://arxiv.org/abs/2105.01924 · HTML: https://mlsanity.com/paper/2105.01924/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 5 May 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Learning Temporal Regularity in Video Sequences](https://mlsanity.com/paper/1604.04574/) (2016)
- [Tile2Vec: Unsupervised representation learning for spatially distributed data](https://mlsanity.com/paper/1805.02855/) (2018)
- [Towards Corner Case Detection for Autonomous Driving](https://mlsanity.com/paper/1902.09184/) (2019)
- [Metric Learning for Novelty and Anomaly Detection](https://mlsanity.com/paper/1808.05492/) (2018)
- [An Unsupervised Random Forest Clustering Technique for Automatic Traffic Scenario Categorization](https://mlsanity.com/paper/2004.02121/) (2020)
- [A Deep Learning Framework for Generation and Analysis of Driving Scenario Trajectories](https://mlsanity.com/paper/2007.14524/) (2020)
- [Clustering of Driving Encounter Scenarios Using Connected Vehicle Trajectories](https://mlsanity.com/paper/1807.08415/) (2018)
- [Unsupervised and Supervised Learning with the Random Forest Algorithm for Traffic Scenario Clustering and Classification](https://mlsanity.com/paper/2004.02126/) (2020)
- [Deep Representation Learning and Clustering of Traffic Scenarios](https://mlsanity.com/paper/2007.07740/) (2020)
- [An Entropy Based Outlier Score and its Application to Novelty Detection for Road Infrastructure Images](https://mlsanity.com/paper/2005.13288/) (2020)

## Built on by (most cited first, 4 of 4)

- [Expert-LaSTS: Expert-Knowledge Guided Latent Space for Traffic Scenarios](https://mlsanity.com/paper/2207.09120/) (2022)
- [ExAgt: Expert-guided Augmentation for Representation Learning of Traffic Scenarios](https://mlsanity.com/paper/2207.08609/) (2022)
- [Conditioning Latent-Space Clusters for Real-World Anomaly Classification](https://mlsanity.com/paper/2309.09676/) (2023)
- [Online Monitoring Framework for Automotive Time Series Data using JEPA Embeddings](https://mlsanity.com/paper/2602.09985/) (2026)

## Abstract (arXiv)

Detecting unknown and untested scenarios is crucial for scenario-based testing. Scenario-based testing is considered to be a possible approach to validate autonomous vehicles. A traffic scenario consists of multiple components, with infrastructure being one of it. In this work, a method to detect novel traffic scenarios based on their infrastructure images is presented. An autoencoder triplet network provides latent representations for infrastructure images which are used for outlier detection. The triplet training of the network is based on the connectivity graphs of the infrastructure. By using the proposed architecture, expert-knowledge is used to shape the latent space such that it incorporates a pre-defined similarity in the neighborhood relationships of an autoencoder. An ablation study on the architecture is highlighting the importance of the triplet autoencoder combination. The best performing architecture is based on vision transformers, a convolution-free attention-based network. The presented method outperforms other state-of-the-art outlier detection approaches.