# Learning richness modulates equality reasoning in neural networks arXiv: https://arxiv.org/abs/2503.09781 ยท HTML: https://mlsanity.com/paper/2503.09781/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Builds on 31 of its 73 references that are in the map. - First posted on arXiv 12 Mar 2025. ## Builds on (most cited first, 30 of 31) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025) - [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023) - [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018) - [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018) - [signSGD: Compressed Optimisation for Non-Convex Problems](https://mlsanity.com/paper/1802.04434/) (2018) - [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018) - [A simple neural network module for relational reasoning](https://mlsanity.com/paper/1706.01427/) (2017) - [Kernel and Rich Regimes in Overparametrized Models](https://mlsanity.com/paper/2002.09277/) (2020) - [Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks](https://mlsanity.com/paper/2302.08399/) (2023) - [Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer](https://mlsanity.com/paper/2203.03466/) (2022) - [Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss](https://mlsanity.com/paper/2002.04486/) (2020) - [The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence](https://mlsanity.com/paper/2002.06177/) (2020) - [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023) - [Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel](https://mlsanity.com/paper/1810.05369/) (2018) - [Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup](https://mlsanity.com/paper/1906.08632/) (2019) - [Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach](https://mlsanity.com/paper/1805.00915/) (2018) - [Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networks](https://mlsanity.com/paper/2205.09653/) (2022) - [Emergent Symbols through Binding in External Memory](https://mlsanity.com/paper/2012.14601/) (2020) - [Feature emergence via margin maximization: case studies in algebraic tasks](https://mlsanity.com/paper/2311.07568/) (2023) - [The Relational Bottleneck as an Inductive Bias for Efficient Abstraction](https://mlsanity.com/paper/2309.06629/) (2023) - [When can transformers reason with abstract symbols?](https://mlsanity.com/paper/2310.09753/) (2023) - [Knowledge Matters: Importance of Prior Information for Optimization](https://mlsanity.com/paper/1301.4083/) (2013) - [Relational reasoning and generalization using non-symbolic neural networks](https://mlsanity.com/paper/2006.07968/) (2020) - [Understanding the computational demands underlying visual reasoning](https://mlsanity.com/paper/2108.03603/) (2021) - [MLPs Learn In-Context on Regression and Classification Tasks](https://mlsanity.com/paper/2405.15618/) (2024) - [The Optimization Landscape of SGD Across the Feature Learning Strength](https://mlsanity.com/paper/2410.04642/) (2024) ## Built on by (most cited first, 2 of 2) - [Boule or Baguette? A Study on Task Topology, Length Generalization, and the Benefit of Reasoning Traces](https://mlsanity.com/paper/2602.14404/) (2026) - [A mathematical theory of balancing relational generalization and memorization](https://mlsanity.com/paper/2605.22972/) (2026) ## Abstract (arXiv) Equality reasoning is ubiquitous and purely abstract: sameness or difference may be evaluated no matter the nature of the underlying objects. As a result, same-different (SD) tasks have been extensively studied as a starting point for understanding abstract reasoning in humans and across animal species. With the rise of neural networks that exhibit striking apparent proficiency for abstractions, equality reasoning in these models has also gained interest. Yet despite extensive study, conclusions about equality reasoning vary widely and with little consensus. To clarify the underlying principles in learning SD tasks, we develop a theory of equality reasoning in multi-layer perceptrons (MLP). Following observations in comparative psychology, we propose a spectrum of behavior that ranges from conceptual to perceptual outcomes. Conceptual behavior is characterized by task-specific representations, efficient learning, and insensitivity to spurious perceptual details. Perceptual behavior is characterized by strong sensitivity to spurious perceptual details, accompanied by the need for exhaustive training to learn the task. We develop a mathematical theory to show that an MLP's behavior is driven by learning richness. Rich-regime MLPs exhibit conceptual behavior, whereas lazy-regime MLPs exhibit perceptual behavior. We validate our theoretical findings in vision SD experiments, showing that rich feature learning promotes success by encouraging hallmarks of conceptual behavior. Overall, our work identifies feature learning richness as a key parameter modulating equality reasoning, and suggests that equality reasoning in humans and animals may similarly depend on learning richness in neural circuits.