mlsanityData to 7 Oct 2026

Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Models

arXiv 2602.01167 ↗First posted 1 Feb 2026cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Current VLMs have demonstrated capabilities across a wide range of multimodal tasks. Typically, in a pretrained VLM, all layers are engaged by default to make predictions on downstream tasks. We find that intervening on a single layer, such as by zeroing its parameters, can improve the performance on certain tasks, indicating that some layers hinder rather than help downstream tasks. We systematically investigate how individual layers influence different tasks via layer intervention. Specifically, we measure the change in performance relative to the base model after intervening on each layer and observe improvements when bypassing specific layers. This improvement can be generalizable across models and datasets, indicating the presence of Task-Interfering Layers that harm downstream tasks' performance. We introduce Task-Layer Interaction Vector, which quantifies the effect of intervening on each layer of a VLM given a task. These task-interfering layers exhibit task-specific sensitivity patterns: tasks requiring similar capabilities show consistent response trends under layer interventions, as evidenced by the high similarity in their task-layer interaction vectors. Inspired by these findings, we propose TaLo (Task-Adaptive Layer Knockout), a training-free, test-time adaptation method that dynamically identifies and bypasses the most interfering layer for a given task. Without parameter updates, TaLo improves performance across various models and datasets, including boosting Qwen-VL's accuracy on the Maps task in ScienceQA by up to 16.6%. Our work reveals an unexpected form of modularity in pretrained VLMs and provides a plug-and-play, training-free mechanism to unlock hidden capabilities at inference time. The source code will be publicly available.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 59 references that are in the map. First posted on arXiv 1 Feb 2026.

Builds on

53 of 59 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Bring Reason to Vision: Understanding Perception and Reasoning through Model Merging8 May
  2. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 50

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  9. 2022Editing Models with Task Arithmeticcited by 1,076
  10. 2022Mass-Editing Memory in a Transformercited by 976
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  12. 2023A Survey on Multimodal Large Language Modelscited by 898
38 more
  1. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  2. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  3. 2021Editing Factual Knowledge in Language Modelscited by 548
  4. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  5. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  6. 2021Fast Model Editing at Scalecited by 491
  7. 2022Memory-Based Model Editing at Scalecited by 443
  8. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  9. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 427
  10. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 379
  11. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 376
  12. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  13. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  14. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  15. 2020Modifying Memories in Transformer Modelscited by 222
  16. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  17. 2023Controlling Text-to-Image Diffusion by Orthogonal Finetuningcited by 207
  18. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  19. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  20. 2023Diverse Data Augmentation with Diffusions for Effective Test-time Prompt Tuningcited by 150
  21. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 142
  22. 2024Efficient Test-Time Adaptation of Vision-Language Modelscited by 138
  23. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  24. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 112
  25. 2025The Curse of Depth in Large Language Modelscited by 58
  26. 2024Investigating Layer Importance in Large Language Modelscited by 35
  27. 2024A deeper look at depth pruning of LLMscited by 23
  28. 2024VLM-Auto: VLM-based Autonomous Driving Assistant with Human-like Behavior and Understanding for Complex Road Scenescited by 22
  29. 2020An Image Enhancing Pattern-based Sparsity for Real-time Inference on Mobile Devicescited by 18
  30. 2024Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levelscited by 16
  31. 2025Transformer-Squared: Self-adaptive LLMscited by 16
  32. 2023Temporal Coherent Test-Time Optimization for Robust Video Classificationcited by 15
  33. 2024Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEditcited by 14
  34. 2025SkipGPT: Dynamic Layer Pruning Reinvented with Token Awareness and Module Decouplingcited by 11
  35. 2025Medical Large Vision Language Models with Multi-Image Visual Abilitycited by 8
  36. 2023Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Modelscited by 7
  37. 2022Data-Efficient Learning via Minimizing Hyperspherical Energycited by 4
  38. 2025Detecting and Pruning Prominent but Detrimental Neurons in Large Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Correcting Visual Blur Induced by Attention Distraction to Reduce Hallucinations: Algorithm and Theorycited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding Task Transfer in Vision-Language Models
  2. 2024Vision-Language Models Create Cross-Modal Task Representations
  3. 2025Skip-It? Theoretical Conditions for Layer Skipping in Vision-Language Models
  4. 2026The Layer Mystery of VLA: An Information-Theoretical Analysis of VLA Latent Interface
  5. 2026Revisiting Parameter Redundancy in Vision-Language-Action Models: Insights from VLM-to-VLA Adaptation
  6. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs
  7. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models
  8. 2025V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models