# Emergence of Shape Bias in Convolutional Neural Networks through Activation Sparsity

arXiv: https://arxiv.org/abs/2310.18894 · HTML: https://mlsanity.com/paper/2310.18894/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 39 references that are in the map.
- First posted on arXiv 29 Oct 2023.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet](https://mlsanity.com/paper/1904.00760/) (2019)
- [Learning De-biased Representations with Biased Representations](https://mlsanity.com/paper/1910.02806/) (2019)
- [Partial success in closing the gap between human and machine vision](https://mlsanity.com/paper/2106.07411/) (2021)
- [Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis](https://mlsanity.com/paper/2101.04775/) (2021)
- [Is Attention Better Than Matrix Decomposition?](https://mlsanity.com/paper/2109.04553/) (2021)
- [Are Adversarial Robustness and Common Perturbation Robustness Independent Attributes ?](https://mlsanity.com/paper/1909.02436/) (2019)
- [Achieving Adversarial Robustness via Sparsity](https://mlsanity.com/paper/2009.05423/) (2020)
- [Visual Concepts and Compositional Voting](https://mlsanity.com/paper/1711.04451/) (2017)

## Built on by (most cited first, 11 of 11)

- [MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation](https://mlsanity.com/paper/2503.20384/) (2025)
- [Toward a Holistic Evaluation of Robustness in CLIP Models](https://mlsanity.com/paper/2410.01534/) (2024)
- [Dissecting Generalized Category Discovery: Multiplex Consensus under Self-Deconstruction](https://mlsanity.com/paper/2508.10731/) (2025)
- [Decomposing the Neurons: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptation](https://mlsanity.com/paper/2405.16486/) (2024)
- [Perceptual Inductive Bias Is What You Need Before Contrastive Learning](https://mlsanity.com/paper/2506.01201/) (2025)
- [Visual Anagrams Reveal Hidden Differences in Holistic Shape Processing Across Vision Models](https://mlsanity.com/paper/2507.00493/) (2025)
- [SAMSOD: Rethinking SAM Optimization for RGB-T Salient Object Detection](https://mlsanity.com/paper/2510.03689/) (2025)
- [On the Reliability of Cue Conflict and Beyond](https://mlsanity.com/paper/2603.10834/) (2026)
- [The Role of Recurrency in Image Segmentation for Noisy and Limited Sample Settings](https://mlsanity.com/paper/2412.15734/) (2024)
- [ShapeY: A Principled Framework for Measuring Shape Recognition Capacity via Nearest-Neighbor Matching](https://mlsanity.com/paper/2604.25065/) (2026)
- [MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation](https://mlsanity.com/paper/2605.17743/) (2026)

## Abstract (arXiv)

Current deep-learning models for object recognition are known to be heavily biased toward texture. In contrast, human visual systems are known to be biased toward shape and structure. What could be the design principles in human visual systems that led to this difference? How could we introduce more shape bias into the deep learning models? In this paper, we report that sparse coding, a ubiquitous principle in the brain, can in itself introduce shape bias into the network. We found that enforcing the sparse coding constraint using a non-differential Top-K operation can lead to the emergence of structural encoding in neurons in convolutional neural networks, resulting in a smooth decomposition of objects into parts and subparts and endowing the networks with shape bias. We demonstrated this emergence of shape bias and its functional benefits for different network structures with various datasets. For object recognition convolutional neural networks, the shape bias leads to greater robustness against style and pattern change distraction. For the image synthesis generative adversary networks, the emerged shape bias leads to more coherent and decomposable structures in the synthesized images. Ablation studies suggest that sparse codes tend to encode structures, whereas the more distributed codes tend to favor texture. Our code is host at the github repository: urlhttps://github.com/Crazy-Jack/nips2023shapevstexture