mlsanityData to 14 Oct 2026

Beyond the Linear Separability Ceiling: Aligning Representations in VLMs

arXiv 2507.07574 ↗First posted 10 Jul 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

A challenge in advancing Visual-Language Models (VLMs) is determining whether their failures on abstract reasoning tasks, such as Bongard problems, stem from flawed perception or faulty top-down reasoning. To disentangle these factors, we introduce a diagnostic framework centered on the Linear Separability Ceiling (LSC), the performance achievable by a linear classifier on a VLM's raw visual embeddings. Applying this framework to state-of-the-art VLMs, we uncover a pervasive ''alignment gap'', where most models fail to generatively outperform the linear separability of their representations. We find that the few models surpassing this ceiling do so via two mechanisms: by further refining visual representations into a more linearly separable format or by executing non-linear decision logic. We demonstrate that this bottleneck is not a fundamental limitation but a solvable visual alignment issue. Our method augments standard next-token prediction with a contrastive objective to restructure the visual manifold into a more one-dimensionally linear geometry, improving image-to-image comparison and enabling models to significantly surpass the LSC on abstract compositional reasoning tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 86 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

64 of 86 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging8 May
  2. 2025Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images19 Feb
  3. 2025A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMs23 Jan
  4. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?5 Jan
  5. 2024VladVA: Discriminative Fine-tuning of LVLMs5 Dec
  6. 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Models4 Dec
  7. 2024Reasoning Limitations of Multimodal Large Language Models. A Case Study of Bongard Problems2 Nov
  8. 2024Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?25 Oct
  9. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation24 Jun
  10. 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns20 Mar
  11. 2023Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning12 Jul
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
1 more
  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  9. 2025Qwen2.5-VL Technical Reportcited by 5,587
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  5. 2025Gemma 3 Technical Reportcited by 1,824
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  8. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  9. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  10. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  11. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  12. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  13. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  14. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  15. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  16. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 226
  17. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  18. 2021CAT: Cross Attention in Vision Transformercited by 92
  19. 2022Text-Only Training for Image Captioning using Noise-Injected CLIPcited by 91
  20. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 67
  21. 2025Scaling Laws for Native Multimodal Modelscited by 50
  22. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
  23. 2025Grokking at the Edge of Numerical Stabilitycited by 46
  24. 2022Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulationscited by 44
  25. 2022A Benchmark for Compositional Visual Reasoningcited by 43
  26. 2022Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactionscited by 34
  27. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  28. 2024Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Datacited by 27
  29. 2019V-PROM: A Benchmark for Visual Reasoning Using Visual Progressive Matricescited by 24
  30. 2024Your Transformer is Secretly Linearcited by 23
  31. 2023Bongard-OpenWorld: Few-Shot Reasoning for Free-form Visual Concepts in the Real Worldcited by 18
  32. 2025PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Modelscited by 16
  33. 2025I-Con: A Unifying Framework for Representation Learningcited by 16
  34. 2025I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?cited by 13
  35. 2024Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Trainingcited by 12
  36. 2025EasyARC: Evaluating Vision Language Models on True Visual Reasoningcited by 10
  37. 2024Pcc-tuning: Breaking the Contrastive Learning Ceiling in Semantic Textual Similaritycited by 5
  38. 2023Support-Set Context Matters for Bongard Problemscited by 3

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Investigating Relational Reasoning in VLMs
  2. 2026Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models
  3. 2026Calibrate Before Reason: Robust Visual Token Reduction against Semantic Drift in VLMs
  4. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs
  5. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability
  6. 2026When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models
  7. 2026Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs
  8. 2026Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap