# Perceptual Inductive Bias Is What You Need Before Contrastive Learning

arXiv: https://arxiv.org/abs/2506.01201 · HTML: https://mlsanity.com/paper/2506.01201/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 1 Jun 2025.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Dense Contrastive Learning for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2011.09157/) (2020)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Partial success in closing the gap between human and machine vision](https://mlsanity.com/paper/2106.07411/) (2021)
- [Video Representation Learning by Dense Predictive Coding](https://mlsanity.com/paper/1909.04656/) (2019)
- [Unsupervised Learning of Dense Visual Representations](https://mlsanity.com/paper/2011.05499/) (2020)
- [Intriguing properties of generative classifiers](https://mlsanity.com/paper/2309.16779/) (2023)
- [TRACER: Extreme Attention Guided Salient Object Tracing Network](https://mlsanity.com/paper/2112.07380/) (2021)
- [On the surprising similarities between supervised and self-supervised models](https://mlsanity.com/paper/2010.08377/) (2020)
- [Emergence of Shape Bias in Convolutional Neural Networks through Activation Sparsity](https://mlsanity.com/paper/2310.18894/) (2023)
- [DenseDINO: Boosting Dense Self-Supervised Learning with Token-Based Point-Level Consistency](https://mlsanity.com/paper/2306.04654/) (2023)

## Built on by (most cited first, 4 of 4)

- [Reallocating Attention Across Layers to Reduce Multimodal Hallucination](https://mlsanity.com/paper/2510.10285/) (2025)
- [On the Reliability of Cue Conflict and Beyond](https://mlsanity.com/paper/2603.10834/) (2026)
- [VINO: Video-driven Invariance for Non-contextual Objects via Structural Prior Guided De-contextualization](https://mlsanity.com/paper/2603.07222/) (2026)
- [Objects Before Words: Object-First Inductive Biases for Grounding Language in Child-View Video](https://mlsanity.com/paper/2606.12985/) (2026)

## Abstract (arXiv)

David Marr's seminal theory of human perception stipulates that visual processing is a multi-stage process, prioritizing the derivation of boundary and surface properties before forming semantic object representations. In contrast, contrastive representation learning frameworks typically bypass this explicit multi-stage approach, defining their objective as the direct learning of a semantic representation space for objects. While effective in general contexts, this approach sacrifices the inductive biases of vision, leading to slower convergence speed and learning shortcut resulting in texture bias. In this work, we demonstrate that leveraging Marr's multi-stage theory-by first constructing boundary and surface-level representations using perceptual constructs from early visual processing stages and subsequently training for object semantics-leads to 2x faster convergence on ResNet18, improved final representations on semantic segmentation, depth estimation, and object recognition, and enhanced robustness and out-of-distribution capability. Together, we propose a pretraining stage before the general contrastive representation pretraining to further enhance the final representation quality and reduce the overall convergence time via inductive bias from human vision systems.