Understanding Task Transfer in Vision-Language Models
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Vision-Language Models (VLMs) perform well on multimodal benchmarks but lag behind humans and specialized models on visual perception tasks like depth estimation or object counting. Finetuning on one task can unpredictably affect performance on others, making task-specific finetuning challenging. In this paper, we address this challenge through a systematic study of task transferability. We examine how finetuning a VLM on one perception task affects its zero-shot performance on others. We introduce Perfection Gap Factor (PGF), a normalized metric that measures change in performance as a result of task transfer. We utilize PGF to compute Task Transferability, which captures both the breadth and the magnitude of transfer induced by a source task. Using three open-weight VLMs evaluated across 13 perception tasks, we construct a task transfer graph that reveals previously unobserved relationships among perception tasks. Our analysis uncovers patterns of positive and negative transfer, identifies groups of tasks that mutually influence each other, organizes tasks into personas based on their transfer behavior and demonstrates how PGF can guide data selection for more efficient training. These findings highlight both opportunities for positive transfer and risks of negative interference, offering actionable guidance for advancing VLMs.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 48 references that are in the map. First posted on arXiv 24 Nov 2025.
Builds on
38 of 48 references.
Closest ancestors · 4
Same direction, newest first
- 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Models4 Dec
- 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases3 Apr
- 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
Other references · 34
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023GPT-4 Technical Reportcited by 18,132
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2023Visual Instruction Tuningcited by 7,792
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2019Towards VQA Models That Can Readcited by 1,696
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
- 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
- 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
22 more
- 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
- 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
- 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
- 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
- 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
- 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
- 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
- 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
- 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
- 2022Visual Spatial Reasoningcited by 288
- 2017HPatches: A benchmark and evaluation of handcrafted and learned local descriptorscited by 261
- 2016Single-Image Depth Perception in the Wildcited by 179
- 2018TallyQA: Answering Complex Counting Questionscited by 152
- 2019SPair-71k: A Large-scale Benchmark for Semantic Correspondencecited by 140
- 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
- 2022Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Datasetcited by 59
- 2022Data-Efficient Finetuning Using Cross-Task Nearest Neighborscited by 26
- 2022There is a Time and Place for Reasoning Beyond the Imagecited by 22
- 2025Sliding Window Attention Training for Efficient Large Language Modelscited by 21
- 2021The Functional Correspondence Problemcited by 20
- 2024When Does Perceptual Alignment Benefit Vision Representations?cited by 18
- 2022Learning More May Not Be Better: Knowledge Transferability in Vision and Language Taskscited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Vision-Language Models Create Cross-Modal Task Representations
- 2026Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Models
- 2025Vision Language Models: A Survey of 26K Papers
- 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models
- 2026Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
- 2025Hidden in plain sight: VLMs overlook their visual representations
- 2026Understanding Cross-Modal Contributions in Continual Vision-Language Models: A Theoretical Perspective
- 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment