# VFA: Vision Frequency Analysis of Foundation Models and Human

arXiv: https://arxiv.org/abs/2409.05817 · HTML: https://mlsanity.com/paper/2409.05817/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 9 Sep 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [CoAtNet: Marrying Convolution and Attention for All Data Sizes](https://mlsanity.com/paper/2106.04803/) (2021)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers](https://mlsanity.com/paper/2208.06366/) (2022)
- [Gradient Matching for Domain Generalization](https://mlsanity.com/paper/2104.09937/) (2021)
- [Generalisation in humans and deep neural networks](https://mlsanity.com/paper/1808.08750/) (2018)
- [Partial success in closing the gap between human and machine vision](https://mlsanity.com/paper/2106.07411/) (2021)
- [MetaFormer Baselines for Vision](https://mlsanity.com/paper/2210.13452/) (2022)
- [Harmonizing the object recognition strategies of deep neural networks with humans](https://mlsanity.com/paper/2211.04533/) (2022)
- [Towards Unsupervised Domain Generalization](https://mlsanity.com/paper/2107.06219/) (2021)
- [Spatial-frequency channels, shape bias, and adversarial robustness](https://mlsanity.com/paper/2309.13190/) (2023)
- [Towards Understanding the Effect of Pretraining Label Granularity](https://mlsanity.com/paper/2303.16887/) (2023)

## Built on by (most cited first, 1 of 1)

- [Online Reasoning Video Segmentation with Just-in-Time Digital Twins](https://mlsanity.com/paper/2503.21056/) (2025)

## Abstract (arXiv)

Machine learning models often struggle with distribution shifts in real-world scenarios, whereas humans exhibit robust adaptation. Models that better align with human perception may achieve higher out-of-distribution generalization. In this study, we investigate how various characteristics of large-scale computer vision models influence their alignment with human capabilities and robustness. Our findings indicate that increasing model and data size and incorporating rich semantic information and multiple modalities enhance models' alignment with human perception and their overall robustness. Our empirical analysis demonstrates a strong correlation between out-of-distribution accuracy and human alignment.