# On the Performance of Concept Probing: The Influence of the Data (Extended Version)

arXiv: https://arxiv.org/abs/2507.18550 · HTML: https://mlsanity.com/paper/2507.18550/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 66 references that are in the map.
- First posted on arXiv 24 Jul 2025.

## Builds on (most cited first, 30 of 38)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [Compositional Explanations of Neurons](https://mlsanity.com/paper/2006.14032/) (2020)
- [What do Neural Machine Translation Models Learn about Morphology?](https://mlsanity.com/paper/1704.03471/) (2017)
- [Debugging Tests for Model Explanations](https://mlsanity.com/paper/2011.05429/) (2020)
- [Concept-based Explainable Artificial Intelligence: A Survey](https://mlsanity.com/paper/2312.12936/) (2023)
- [Analyzing Individual Neurons in Pre-trained Language Models](https://mlsanity.com/paper/2010.02695/) (2020)
- [A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts](https://mlsanity.com/paper/2105.07190/) (2021)
- [Low-Complexity Probing via Finding Subnetworks](https://mlsanity.com/paper/2104.03514/) (2021)
- [There and Back Again: Revisiting Backpropagation Saliency Methods](https://mlsanity.com/paper/2004.02866/) (2020)
- [Concept Activation Regions: A Generalized Framework For Concept-Based Explanations](https://mlsanity.com/paper/2209.11222/) (2022)
- [Pareto Probing: Trading Off Accuracy for Complexity](https://mlsanity.com/paper/2010.02180/) (2020)
- [Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation](https://mlsanity.com/paper/2207.06214/) (2022)
- [How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labels](https://mlsanity.com/paper/2008.11721/) (2020)
- [What if This Modified That? Syntactic Interventions via Counterfactual Embeddings](https://mlsanity.com/paper/2105.14002/) (2021)
- [Machine Learning Model Sizes and the Parameter Gap](https://mlsanity.com/paper/2207.02852/) (2022)
- [Visual correspondence-based explanations improve AI robustness and human-AI team accuracy](https://mlsanity.com/paper/2208.00780/) (2022)

## Abstract (arXiv)

Concept probing has recently garnered increasing interest as a way to help interpret artificial neural networks, dealing both with their typically large size and their subsymbolic nature, which ultimately renders them unfeasible for direct human interpretation. Concept probing works by training additional classifiers to map the internal representations of a model into human-defined concepts of interest, thus allowing humans to peek inside artificial neural networks. Research on concept probing has mainly focused on the model being probed or the probing model itself, paying limited attention to the data required to train such probing models. In this paper, we address this gap. Focusing on concept probing in the context of image classification tasks, we investigate the effect of the data used to train probing models on their performance. We also make available concept labels for two widely used datasets.