# Examining CNN Representations with respect to Dataset Bias

arXiv: https://arxiv.org/abs/1710.10577 · HTML: https://mlsanity.com/paper/1710.10577/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2017.

## Built on by (most cited first, 27 of 27)

- [Taxonomy of Machine Learning Safety: A Survey and Primer](https://mlsanity.com/paper/2106.04823/) (2021)
- [Concept Embedding Analysis: A Review](https://mlsanity.com/paper/2203.13909/) (2022)
- [Distilling Cognitive Backdoor Patterns within an Image](https://mlsanity.com/paper/2301.10908/) (2023)
- [Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?](https://mlsanity.com/paper/2409.13456/) (2024)
- [IFBiD: Inference-Free Bias Detection](https://mlsanity.com/paper/2109.04374/) (2021)
- [Evaluating the Stability of Semantic Concept Representations in CNNs for Robust Explainability](https://mlsanity.com/paper/2304.14864/) (2023)
- [Discovering and Explaining the Non-Causality of Deep Learning in SAR ATR](https://mlsanity.com/paper/2304.00668/) (2023)
- [Zone Evaluation: Revealing Spatial Bias in Object Detection](https://mlsanity.com/paper/2310.13215/) (2023)
- [Local Concept Embeddings for Analysis of Concept Distributions in Vision DNN Feature Spaces](https://mlsanity.com/paper/2311.14435/) (2023)
- [Towards Measuring Bias in Image Classification](https://mlsanity.com/paper/2107.00360/) (2021)
- [Through a fair looking-glass: mitigating bias in image datasets](https://mlsanity.com/paper/2209.08648/) (2022)
- [BAdd: Bias Mitigation through Bias Addition](https://mlsanity.com/paper/2408.11439/) (2024)
- [Unsupervised Domain Expansion for Visual Categorization](https://mlsanity.com/paper/2104.00233/) (2021)
- [Efficient Human-in-the-loop System for Guiding DNNs Attention](https://mlsanity.com/paper/2206.05981/) (2022)
- [Exploring the Impact of Dataset Bias on Dataset Distillation](https://mlsanity.com/paper/2403.16028/) (2024)
- [Challenges and Opportunities for Machine Learning Classification of Behavior and Mental State from Images](https://mlsanity.com/paper/2201.11197/) (2022)
- [Concept Evolution in Deep Learning Training: A Unified Interpretation Framework and Discoveries](https://mlsanity.com/paper/2203.16475/) (2022)
- [Unraveling Machine Behavior by Multi-Level Bias Analysis and Detection: Methodology and Application to Computer Vision](https://mlsanity.com/paper/2607.07236/) (2026)
- [Learning Interpretable Concept Groups in CNNs](https://mlsanity.com/paper/2109.10078/) (2021)
- [Improving Network Interpretability via Explanation Consistency Evaluation](https://mlsanity.com/paper/2408.04600/) (2024)
- [FaceX: Understanding Face Attribute Classifiers through Summary Model Explanations](https://mlsanity.com/paper/2412.07313/) (2024)
- [Faithful Counterfactual Visual Explanations (FCVE)](https://mlsanity.com/paper/2501.06841/) (2025)
- [Interpreting Hidden Semantics in the Intermediate Layers of 3D Point Cloud Classification Neural Network](https://mlsanity.com/paper/2303.06652/) (2023)
- [Extracting Human Attention through Crowdsourced Patch Labeling](https://mlsanity.com/paper/2403.15013/) (2024)
- [A Neurosymbolic Framework for Bias Correction in Convolutional Neural Networks](https://mlsanity.com/paper/2405.15886/) (2024)
- [Towards Counterfactual and Contrastive Explainability and Transparency of DCNN Image Classifiers](https://mlsanity.com/paper/2501.06831/) (2025)
- [Gradient-based Model Shortcut Detection for Time Series Classification](https://mlsanity.com/paper/2510.10075/) (2025)

## Abstract (arXiv)

Given a pre-trained CNN without any testing samples, this paper proposes a simple yet effective method to diagnose feature representations of the CNN. We aim to discover representation flaws caused by potential dataset bias. More specifically, when the CNN is trained to estimate image attributes, we mine latent relationships between representations of different attributes inside the CNN. Then, we compare the mined attribute relationships with ground-truth attribute relationships to discover the CNN's blind spots and failure modes due to dataset bias. In fact, representation flaws caused by dataset bias cannot be examined by conventional evaluation strategies based on testing images, because testing images may also have a similar bias. Experiments have demonstrated the effectiveness of our method.