# Do computer vision foundation models learn the low-level characteristics of the human visual system?

arXiv: https://arxiv.org/abs/2502.20256 · HTML: https://mlsanity.com/paper/2502.20256/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 44 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Partial success in closing the gap between human and machine vision](https://mlsanity.com/paper/2106.07411/) (2021)
- [Are Convolutional Neural Networks or Transformers more like human vision?](https://mlsanity.com/paper/2105.07197/) (2021)
- [Neither hype nor gloom do DNNs justice](https://mlsanity.com/paper/2312.05355/) (2023)
- [ColorVideoVDP: A visual difference predictor for image, video and display distortions](https://mlsanity.com/paper/2401.11485/) (2024)
- [Generalizable Black-Box Adversarial Attack with Meta Learning](https://mlsanity.com/paper/2301.00364/) (2023)
- [Contrast Sensitivity Functions in Autoencoders](https://mlsanity.com/paper/2103.00481/) (2021)
- [BridgeNet: Comprehensive and Effective Feature Interactions via Bridge Feature for Multi-task Dense Predictions](https://mlsanity.com/paper/2312.13514/) (2023)
- [Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics](https://mlsanity.com/paper/2502.11725/) (2025)
- [Multi-Object Manipulation via Object-Centric Neural Scattering Functions](https://mlsanity.com/paper/2306.08748/) (2023)
- [Foundation Models Boost Low-Level Perceptual Similarity Metrics](https://mlsanity.com/paper/2409.07650/) (2024)
- [Rethinking Cross-Domain Pedestrian Detection: A Background-Focused Distribution Alignment Framework for Instance-Free One-Stage Detectors](https://mlsanity.com/paper/2309.08771/) (2023)

## Built on by (most cited first, 12 of 12)

- [Toward Generalizable Forgery Detection and Reasoning](https://mlsanity.com/paper/2503.21210/) (2025)
- [Hallucination Score: Towards Mitigating Hallucinations in Generative Image Super-Resolution](https://mlsanity.com/paper/2507.14367/) (2025)
- [UniSER: A Foundation Model for Unified Soft Effects Removal](https://mlsanity.com/paper/2511.14183/) (2025)
- [Evaluating quality metrics through the lenses of psychophysical measurements of low-level vision](https://mlsanity.com/paper/2503.16264/) (2025)
- [Contrast Sensitivity in Multimodal Large Language Models: A Psychophysics-Inspired Evaluation](https://mlsanity.com/paper/2508.10367/) (2025)
- [EDITS: Enhancing Dataset Distillation with Implicit Textual Semantics](https://mlsanity.com/paper/2509.13858/) (2025)
- [Do Vision Encoders Exhibit Human-like Color Thresholds?](https://mlsanity.com/paper/2607.16540/) (2026)
- [Break Stylistic Sophon: Are We Really Meant to Confine the Imagination in Style Transfer?](https://mlsanity.com/paper/2506.15033/) (2025)
- [Metric-Guided Feature Fusion of Visual Foundation Models for Segmentation Tasks](https://mlsanity.com/paper/2605.16864/) (2026)
- [The Platonic Defense: Backdoor Defense for Self-Supervised Encoders in the Era of Large Scale Pre-training](https://mlsanity.com/paper/2606.29451/) (2026)
- [IRIS: A Visual Cortex-Inspired Framework for Analyzing Orientation Selectivity in Vision Transformers](https://mlsanity.com/paper/2608.05122/) (2026)
- [BTC3D: Blended Tile Conditioning for Detail-Enhancing Image-to-3D Generation](https://mlsanity.com/paper/2609.39709/) (2026)

## Abstract (arXiv)

Computer vision foundation models, such as DINO or OpenCLIP, are trained in a self-supervised manner on large image datasets. Analogously, substantial evidence suggests that the human visual system (HVS) is influenced by the statistical distribution of colors and patterns in the natural world, characteristics also present in the training data of foundation models. The question we address in this paper is whether foundation models trained on natural images mimic some of the low-level characteristics of the human visual system, such as contrast detection, contrast masking, and contrast constancy. Specifically, we designed a protocol comprising nine test types to evaluate the image encoders of 45 foundation and generative models. Our results indicate that some foundation models (e.g., DINO, DINOv2, and OpenCLIP), share some of the characteristics of human vision, but other models show little resemblance. Foundation models tend to show smaller sensitivity to low contrast and rather irregular responses to contrast across frequencies. The foundation models show the best agreement with human data in terms of contrast masking. Our findings suggest that human vision and computer vision may take both similar and different paths when learning to interpret images of the real world. Overall, while differences remain, foundation models trained on vision tasks start to align with low-level human vision, with DINOv2 showing the closest resemblance.