mlsanityData to 14 Oct 2026

Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models

arXiv 2502.15678 ↗First posted 21 Feb 2025cs.LGcited by 4

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Pre-trained vision language models still fall short of human visual cognition. In an effort to improve visual cognition and align models with human behavior, we introduce visual stimuli and human judgments on visual cognition tasks, allowing us to systematically evaluate performance across cognitive domains under a consistent environment. We fine-tune models on ground truth data for intuitive physics and causal reasoning and find that this improves model performance in the respective fine-tuning domain. Furthermore, it can improve model alignment with human behavior. However, we find that task-specific fine-tuning does not contribute to robust human-like generalization to data with other visual characteristics or to tasks in other cognitive domains.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 57 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

29 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024The Illusion-Illusion: Vision Language Models See Illusions Where There Are None7 Dec
  2. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem31 Oct
  3. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  4. 2024Vision language models are blind: Failing to translate detailed visual features into words9 Jul
  5. 2023Visual cognition in multimodal large language models27 Nov
  6. 2023Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?31 Oct

Other references · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  8. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  9. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  10. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 656
  11. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  12. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
11 more
  1. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  2. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
  3. 2018Generalisation in humans and deep neural networkscited by 237
  4. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 199
  5. 2023CLadder: Assessing Causal Reasoning in Language Modelscited by 107
  6. 2020Beyond accuracy: quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistencycited by 78
  7. 2016Learning Physical Intuition of Block Towers by Examplecited by 68
  8. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  9. 2022Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning taskscited by 56
  10. 2023How Does Fine-Tuning Impact Out-of-Distribution Detection for Vision-Language Models?cited by 38
  11. 2024Centaur: a foundation model of human cognitioncited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Vision Language Models Cannot Reason About Physical Transformationcited by 4
  2. 2026Chart-RL: Generalized Chart Comprehension via Reinforcement Learning with Verifiable Rewardscited by 3
  3. 2025InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoningcited by 2
  4. 2026Can Vision Language Models Learn Intuitive Physics from Interaction?cited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs
  2. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Models
  3. 2026Reasoning Dynamics and the Limits of Monitoring Modality Reliance in Vision-Language Models
  4. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training
  5. 2026The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models
  6. 2025What's Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning
  7. 2025Visual Language Models show widespread visual deficits on neuropsychological tests
  8. 2026Visual Attention Faithfulness in Vision-Language Models is Heterogeneous