# Novelty-based Generalization Evaluation for Traffic Light Detection

arXiv: https://arxiv.org/abs/2201.00531 · HTML: https://mlsanity.com/paper/2201.00531/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 32 references that are in the map.
- First posted on arXiv 3 Jan 2022.

## Builds on (most cited first, 8 of 8)

- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Understanding disentangling in β-VAE](https://mlsanity.com/paper/1804.03599/) (2018)
- [PyOD: A Python Toolbox for Scalable Outlier Detection](https://mlsanity.com/paper/1901.01588/) (2019)
- [Robust Anomaly Detection in Images using Adversarial Autoencoders](https://mlsanity.com/paper/1901.06355/) (2019)
- [VATLD: A Visual Analytics System to Assess, Understand and Improve Traffic Light Detection](https://mlsanity.com/paper/2009.12975/) (2020)
- [q-Space Novelty Detection with Variational Autoencoders](https://mlsanity.com/paper/1806.02997/) (2018)
- [A comparative evaluation of novelty detection algorithms for discrete sequences](https://mlsanity.com/paper/1902.10940/) (2019)
- [Generalization ability of region proposal networks for multispectral person detection](https://mlsanity.com/paper/1905.02758/) (2019)

## Abstract (arXiv)

The advent of Convolutional Neural Networks (CNNs) has led to their application in several domains. One noteworthy application is the perception system for autonomous driving that relies on the predictions from CNNs. Practitioners evaluate the generalization ability of such CNNs by calculating various metrics on an independent test dataset. A test dataset is often chosen based on only one precondition, i.e., its elements are not a part of the training data. Such a dataset may contain objects that are both similar and novel w.r.t. the training dataset. Nevertheless, existing works do not reckon the novelty of the test samples and treat them all equally for evaluating generalization. Such novelty-based evaluations are of significance to validate the fitness of a CNN in autonomous driving applications. Hence, we propose a CNN generalization scoring framework that considers novelty of objects in the test dataset. We begin with the representation learning technique to reduce the image data into a low-dimensional space. It is on this space we estimate the novelty of the test samples. Finally, we calculate the generalization score as a combination of the test data prediction performance and novelty. We perform an experimental study of the same for our traffic light detection application. In addition, we systematically visualize the results for an interpretable notion of novelty.