mlsanityData to 14 Oct 2026

Birds of a Feather Flock Together: Background-Invariant Representations via Linear Structure in VLMs

arXiv 2605.11107 ↗First posted 11 May 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs), such as CLIP and SigLIP 2, are widely used for image classification, yet their vision encoders remain vulnerable to systematic biases that undermine robustness. In particular, correlations between foreground objects and their backgrounds constitute a salient and practically important class of spurious dependencies. In this work, we revisit the well-known property of high linear additivity in VLM embedding spaces and show that it enables a decomposition of scene representations into foreground and background components. Leveraging this insight, we introduce a pre-training approach that exploits this property to construct background-invariant representations using synthetic data. Our method achieves, to our knowledge, the first worst-group accuracy exceeding 90% on Waterbirds under perfect (100%) spurious correlation (i.e., no minority-group examples in the training data). Furthermore, it demonstrates strong sim-to-real transfer and requires no access to real-world debiased data, making it practical for real-world deployment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 43 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

35 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Interpreting the linear structure of vision-language model embedding spaces16 Apr
  2. 2024RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models6 Nov
  3. 2023Vision-Language Models for Vision Tasks: A Survey3 Apr

Other references · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023Segment Anythingcited by 8,234
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  11. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  12. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
20 more
  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  2. 2021Transformers in Vision: A Surveycited by 1,013
  3. 2018Robustness May Be at Odds with Accuracycited by 880
  4. 2021Robust fine-tuning of zero-shot modelscited by 797
  5. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  6. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  7. 2021ImageNet-21K Pretraining for the Massescited by 613
  8. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  9. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  10. 2023Demystifying CLIP Datacited by 270
  11. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 119
  12. 2022NICO++: Towards Better Benchmarking for Domain Generalizationcited by 86
  13. 2023Simple and Fast Group Robustness by Automatic Feature Reweightingcited by 68
  14. 2025Interpreting CLIP with Hierarchical Sparse Autoencoderscited by 60
  15. 2024A Sober Look at the Robustness of CLIPs to Spurious Featurescited by 38
  16. 2023Zero-Shot Robustification of Zero-Shot Modelscited by 24
  17. 2024Out of Spuriousity: Improving Robustness to Spurious Correlations without Group Annotationscited by 9
  18. 2024Bridging Explainability and Embeddings: BEE Aware of Spuriousnesscited by 2
  19. 2025Superclass-Guided Representation Disentanglement for Spurious Correlation Mitigationcited by 2
  20. 2025Robust Context-Aware Object Recognitioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Breaking Spurious Correlations via Generative Randomization and Cross-Variant Self-Supervised Learning
  2. 2026Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance
  3. 2026Revisiting Visual Representation Enhancement of VLMs via Kernel Canonical Correlation Analysis
  4. 2024Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions
  5. 2025Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere
  6. 2024BendVLM: Test-Time Debiasing of Vision-Language Embeddings
  7. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs
  8. 2026Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders