# Network Generalization Prediction for Safety Critical Tasks in Novel Operating Domains

arXiv: https://arxiv.org/abs/2108.07399 · HTML: https://mlsanity.com/paper/2108.07399/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 17 Aug 2021.

## Builds on (most cited first, 16 of 16)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://mlsanity.com/paper/2002.11297/) (2020)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [A Review of Feature Selection Methods Based on Mutual Information](https://mlsanity.com/paper/1509.07577/) (2015)
- [Scalable Greedy Feature Selection via Weak Submodularity](https://mlsanity.com/paper/1703.02723/) (2017)
- [Towards Dependability Metrics for Neural Networks](https://mlsanity.com/paper/1806.02338/) (2018)
- [Info-Greedy sequential adaptive compressed sensing](https://mlsanity.com/paper/1407.0731/) (2014)
- [Greedy-Based Feature Selection for Efficient LiDAR SLAM](https://mlsanity.com/paper/2103.13090/) (2021)
- [Dependable Neural Networks for Safety Critical Tasks](https://mlsanity.com/paper/1912.09902/) (2019)
- [Jigsaw-VAE: Towards Balancing Features in Variational Autoencoders](https://mlsanity.com/paper/2005.05496/) (2020)
- [Evaluating Model Robustness and Stability to Dataset Shift](https://mlsanity.com/paper/2010.15100/) (2020)

## Built on by (most cited first, 2 of 2)

- [Mapping DNN Embedding Manifolds for Network Generalization Prediction](https://mlsanity.com/paper/2202.03868/) (2022)
- [Nearest Neighbor Projection Removal Adversarial Training](https://mlsanity.com/paper/2509.07673/) (2025)

## Abstract (arXiv)

It is well known that Neural Network (network) performance often degrades when a network is used in novel operating domains that differ from its training and testing domains. This is a major limitation, as networks are being integrated into safety critical, cyber-physical systems that must work in unconstrained environments, e.g., perception for autonomous vehicles. Training networks that generalize to novel operating domains and that extract robust features is an active area of research, but previous work fails to predict what the network performance will be in novel operating domains. We propose the task Network Generalization Prediction: predicting the expected network performance in novel operating domains. We describe the network performance in terms of an interpretable Context Subspace, and we propose a methodology for selecting the features of the Context Subspace that provide the most information about the network performance. We identify the Context Subspace for a pretrained Faster RCNN network performing pedestrian detection on the Berkeley Deep Drive (BDD) Dataset, and demonstrate Network Generalization Prediction accuracy within 5% or less of observed performance. We also demonstrate that the Context Subspace from the BDD Dataset is informative for completely unseen datasets, JAAD and Cityscapes, where predictions have a bias of 10% or less.