mlsanityData to 7 Oct 2026

Vision-Language Models Create Cross-Modal Task Representations

arXiv 2410.22330 ↗First posted 29 Oct 2024cs.CV cs.CL cs.LGcited by 14

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Autoregressive vision-language models (VLMs) can handle many tasks within a single model, yet the representations that enable this capability remain opaque. We find that VLMs align conceptually equivalent inputs into a shared task vector, which is invariant to modality (text, image) and format (examples, instruction), and may simplify VLM processing. We measure this alignment via cross-modal transfer -- the ability of a task vector derived in one modality to trigger the correct generation in another -- on a range of tasks and model architectures. Although the task vector is highly compressed, we find that this single vector outperforms prompting the model with the full task information, unique to this cross-modal case. Furthermore, we show that task vectors can be transferred from a base language model to its fine-tuned vision-language counterpart, and that they can be derived solely from instructions without the need for examples. Taken together, our findings shed light on how VLMs internally process task information, and how they map different modalities into common semantic representations. Project page: https://vlm-cross-modal-reps.github.io.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 53 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

35 of 53 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  2. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning21 Jun
  3. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  8. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  9. 2022In-context Learning and Induction Headscited by 930
  10. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  11. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  12. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
20 more
  1. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  2. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  3. 2018Explaining Explanations: An Overview of Interpretability of Machine Learningcited by 508
  4. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  5. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  6. 2024The Platonic Representation Hypothesiscited by 362
  7. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  8. 2024What matters when building vision-language models?cited by 321
  9. 2023Language Models Represent Space and Timecited by 306
  10. 2023In-Context Learning Creates Task Vectorscited by 302
  11. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  12. 2023Function Vectors in Large Language Modelscited by 292
  13. 2013Direct and Indirect Effectscited by 276
  14. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  15. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  16. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  17. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  18. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 110
  19. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  20. 2024Finding Visual Task Vectorscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

  1. 2025Forgotten Polygons: Multimodal Large Language Models are Shape-Blindcited by 16
  2. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Modelscited by 11
  3. 2025Do different prompting methods yield a common task representation in language models?cited by 9
  4. 2026Seeing but Not Thinking: Routing Distraction in Multimodal Mixture-of-Expertscited by 8
  5. 2025Scaling can lead to compositional generalizationcited by 7
  6. 2025Just-in-time and distributed task representations in language modelscited by 7
  7. 2025Cross-Model Transfer of Task Vectors via Few-Shot Orthogonal Alignmentcited by 1
  8. 2026Uncertainty-Aware Multimodal Learning via Conformal Shapley Intervalscited by 1
  9. 2025STEMTOX: From Collaborative Tags to Fine-Grained Toxic Meme Detection via Entropy-Guided Multi-Task Learningcited by 0
  10. 2025Variational Task Vector Compositioncited by 0
  11. 2026Interpreting and Enhancing Emotional Circuits in Large Vision-Language Models via Cross-Modal Information Flowcited by 0
  12. 2026Attending to Multimodal Generation One Token at a Timecited by 0
2 more
  1. 2026The Alignment Illusion in Multimodal Large Language Modelscited by 0
  2. 2026Geometric Similarity in VLM Low-Level Vision Representationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs
  2. 2026Understanding Cross-Modal Contributions in Continual Vision-Language Models: A Theoretical Perspective
  3. 2025Hidden in plain sight: VLMs overlook their visual representations
  4. 2026The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment
  5. 2026Investigating Cross-Modal Skill Injection: Scenarios, Methods, and Hyperparameters
  6. 2022Vision-Language Intelligence: Tasks, Representation Learning, and Large Models
  7. 2025Cross-Task Generalization Between Understanding and Generation in Unified Vision-Language Models: A Controlled Study
  8. 2026Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability