# Probing artificial neural networks: insights from neuroscience

arXiv: https://arxiv.org/abs/2104.08197 · HTML: https://mlsanity.com/paper/2104.08197/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 42 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020)
- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?](https://mlsanity.com/paper/2005.00719/) (2020)
- [Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information](https://mlsanity.com/paper/1808.08079/) (2018)
- [Do Language Embeddings Capture Scales?](https://mlsanity.com/paper/2010.05345/) (2020)
- [Pareto Probing: Trading Off Accuracy for Complexity](https://mlsanity.com/paper/2010.02180/) (2020)
- [Neural population geometry: An approach for understanding biological and artificial neural networks](https://mlsanity.com/paper/2104.07059/) (2021)
- [Do Neural Language Models Show Preferences for Syntactic Formalisms?](https://mlsanity.com/paper/2004.14096/) (2020)
- [Interpreting Encoding and Decoding Models](https://mlsanity.com/paper/1812.00278/) (2018)
- [From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning](https://mlsanity.com/paper/1610.03342/) (2016)

## Built on by (most cited first, 17 of 17)

- [Conditional probing: measuring usable information beyond a baseline](https://mlsanity.com/paper/2109.09234/) (2021)
- [SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysis](https://mlsanity.com/paper/2302.00785/) (2023)
- [Dissociating language and thought in large language models](https://mlsanity.com/paper/2301.06627/) (2023)
- [Language Models as Models of Language](https://mlsanity.com/paper/2408.07144/) (2024)
- [Representation Engineering for Large-Language Models: Survey and Research Challenges](https://mlsanity.com/paper/2502.17601/) (2025)
- [Neural Population Geometry Reveals the Role of Stochasticity in Robust Perception](https://mlsanity.com/paper/2111.06979/) (2021)
- [Probing Graph Representations](https://mlsanity.com/paper/2303.03951/) (2023)
- [From Weights to Activations: Is Steering the Next Frontier of Adaptation?](https://mlsanity.com/paper/2604.14090/) (2026)
- [Trustworthy Social Bias Measurement](https://mlsanity.com/paper/2212.11672/) (2022)
- [Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neuroscience](https://mlsanity.com/paper/2408.12664/) (2024)
- [Vision-Language Models Align with Human Neural Representations in Concept Processing](https://mlsanity.com/paper/2407.17914/) (2024)
- [Prophecy: Inferring Formal Properties from Neuron Activations](https://mlsanity.com/paper/2509.21677/) (2025)
- [Language Cognition and Language Computation -- Human and Machine Language Understanding](https://mlsanity.com/paper/2301.04788/) (2023)
- [Can We Use Probing to Better Understand Fine-tuning and Knowledge Distillation of the BERT NLU?](https://mlsanity.com/paper/2301.11688/) (2023)
- [Rethinking the Construction of Effective Metrics for Understanding the Mechanisms of Pretrained Language Models](https://mlsanity.com/paper/2310.12454/) (2023)
- [RouterKT: Mixture-of-Experts for Knowledge Tracing](https://mlsanity.com/paper/2504.08989/) (2025)
- [Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe](https://mlsanity.com/paper/2605.00607/) (2026)

## Abstract (arXiv)

A major challenge in both neuroscience and machine learning is the development of useful tools for understanding complex information processing systems. One such tool is probes, i.e., supervised models that relate features of interest to activation patterns arising in biological or artificial neural networks. Neuroscience has paved the way in using such models through numerous studies conducted in recent decades. In this work, we draw insights from neuroscience to help guide probing research in machine learning. We highlight two important design choices for probes - direction and expressivity - and relate these choices to research goals. We argue that specific research goals play a paramount role when designing a probe and encourage future probing studies to be explicit in stating these goals.