# Representation biases: will we achieve complete understanding by analyzing representations?

arXiv: https://arxiv.org/abs/2507.22216 · HTML: https://mlsanity.com/paper/2507.22216/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 63 references that are in the map.
- First posted on arXiv 29 Jul 2025.

## Builds on (most cited first, 17 of 17)

- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [On the Origin of Implicit Regularization in Stochastic Gradient Descent](https://mlsanity.com/paper/2101.12176/) (2021)
- [The Low-Rank Simplicity Bias in Deep Networks](https://mlsanity.com/paper/2103.10427/) (2021)
- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [What shapes feature representations? Exploring datasets, architectures, and training](https://mlsanity.com/paper/2006.12433/) (2020)
- [On the Foundations of Shortcut Learning](https://mlsanity.com/paper/2310.16228/) (2023)
- [The Neural Race Reduction: Dynamics of Abstraction in Gated Networks](https://mlsanity.com/paper/2207.10430/) (2022)
- [Harmonics of Learning: Universal Fourier Features Emerge in Invariant Networks](https://mlsanity.com/paper/2312.08550/) (2023)
- [Learned feature representations are biased by complexity, learning order, position, and more](https://mlsanity.com/paper/2405.05847/) (2024)
- [Missed Causes and Ambiguous Effects: Counterfactuals Pose Challenges for Interpreting Neural Networks](https://mlsanity.com/paper/2407.04690/) (2024)
- [Soft Matching Distance: A metric on neural representations that captures single-neuron tuning](https://mlsanity.com/paper/2311.09466/) (2023)
- [What Representational Similarity Measures Imply about Decodable Information](https://mlsanity.com/paper/2411.08197/) (2024)
- [Understanding Visual Feature Reliance through the Lens of Complexity](https://mlsanity.com/paper/2407.06076/) (2024)
- [Brain-Model Evaluations Need the NeuroAI Turing Test](https://mlsanity.com/paper/2502.16238/) (2025)
- [Naturalistic Computational Cognitive Science: Towards generalizable models and theories that capture the full range of natural behavior](https://mlsanity.com/paper/2502.20349/) (2025)
- [Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neuroscience](https://mlsanity.com/paper/2408.12664/) (2024)

## Built on by (most cited first, 6 of 6)

- [Addressing divergent representations from causal interventions on neural networks](https://mlsanity.com/paper/2511.04638/) (2025)
- [The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safety](https://mlsanity.com/paper/2601.06116/) (2026)
- [Stimulus symmetries can confound representational similarity analyses](https://mlsanity.com/paper/2605.21324/) (2026)
- [Bears, all bears, and some bears. Language Constraints on Language Models' Inductive Inferences](https://mlsanity.com/paper/2601.09852/) (2026)
- [Signal-to-Noise Ratio and Sample Size Govern Representational Alignment in Neural Networks](https://mlsanity.com/paper/2605.26973/) (2026)
- [On the use of foundation models in cognitive science](https://mlsanity.com/paper/2608.07812/) (2026)

## Abstract (arXiv)

A common approach in neuroscience is to study neural representations as a means to understand a system -- increasingly, by relating the neural representations to the internal representations learned by computational models. However, a recent work in machine learning (Lampinen, 2024) shows that learned feature representations may be biased to over-represent certain features, and represent others more weakly and less-consistently. For example, simple (linear) features may be more strongly and more consistently represented than complex (highly nonlinear) features. These biases could pose challenges for achieving full understanding of a system through representational analysis. In this perspective, we illustrate these challenges -- showing how feature representation biases can lead to strongly biased inferences from common analyses like PCA, regression, and RSA. We also present homomorphic encryption as a simple case study of the potential for strong dissociation between patterns of representation and computation. We discuss the implications of these results for representational comparisons between systems, and for neuroscience more generally.