mlsanityData to 14 Oct 2026

Finding Visual Task Vectors

arXiv 2404.05729 ↗First posted 8 Apr 2024cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

Visual Prompting is a technique for teaching models to perform a visual task via in-context examples, without any additional training. In this work, we analyze the activations of MAE-VQGAN, a recent Visual Prompting model, and find task vectors, activations that encode task-specific information. Equipped with this insight, we demonstrate that it is possible to identify the task vectors and use them to guide the network towards performing different tasks without providing any input-output examples. To find task vectors, we compute the average intermediate activations per task and use the REINFORCE algorithm to search for the subset of task vectors. The resulting task vectors guide the model towards performing a task better than the original model without the need for input-output examples.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Builds on 42 of its 52 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

42 of 52 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  11. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  12. 2022Visual Prompt Tuningcited by 1,681
30 more
  1. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  3. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  4. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  6. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  7. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  8. 2017One-Shot Learning for Semantic Segmentationcited by 337
  9. 2023In-Context Learning Creates Task Vectorscited by 312
  10. 2023Function Vectors in Large Language Modelscited by 308
  11. 2020A Survey on Neural Network Interpretabilitycited by 291
  12. 2013Direct and Indirect Effectscited by 280
  13. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 274
  14. 2022Visual Prompting via Image Inpaintingcited by 250
  15. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  16. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  17. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 196
  18. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  19. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  20. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 91
  21. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  22. 2020Causal Interpretability for Machine Learning -- Problems, Methods and Evaluationcited by 82
  23. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  24. 2024From Understanding to Utilization: A Survey on Explainability for Large Language Modelscited by 48
  25. 2023How are Prompts Different in Terms of Sensitivity?cited by 35
  26. 2024Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?cited by 24
  27. 2024Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learningcited by 20
  28. 2023IMProv: Inpainting-based Multimodal Prompting for Computer Vision Taskscited by 10
  29. 2024Transformer-based Causal Language Models Perform Clusteringcited by 5
  30. 2023Emergence and Function of Abstract Representations in Self-Supervised Transformerscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 10; 2026: 8.

24 papers.

Sort:

By area:

  1. 2024Perceive-IR: Learning to Perceive Degradation Better for All-in-One Image Restorationcited by 62
  2. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 50
  3. 2024Vision-Language Models Create Cross-Modal Task Representationscited by 15
  4. 2025Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrationscited by 15
  5. 2024WeatherGFM: Learning A Weather Generalist Foundation Model via In-context Learningcited by 12
  6. 2024Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Featurescited by 11
  7. 2025Do different prompting methods yield a common task representation in language models?cited by 11
  8. 2025Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitationscited by 9
  9. 2024ELICIT: LLM Augmentation via External In-Context Capabilitycited by 7
  10. 2025Activation Reward Models for Few-Shot Model Alignmentcited by 6
  11. 2026An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapsecited by 4
  12. 2025Think Outside the Policy: In-Context Steered Policy Optimizationcited by 3
12 more
  1. 2025Towards Ambiguity-Free Spatial Foundation Model: Rethinking and Decoupling Depth Ambiguitycited by 2
  2. 2025Evaluating Steering Techniques using Human Similarity Judgmentscited by 2
  3. 2026PromptHub: Enhancing Multi-Prompt Visual In-Context Learning with Locality-Aware Fusion, Concentration and Alignmentcited by 2
  4. 2025PANICL: Mitigating Over-Reliance on Single Prompt in Visual In-Context Learningcited by 1
  5. 2025Multimodal Function Vectors for Visual Relationscited by 1
  6. 2026Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selectioncited by 1
  7. 2025Variational Task Vector Compositioncited by 0
  8. 2026Taking Shortcuts for Categorical VQA Using Super Neuronscited by 0
  9. 2026Probing Intrinsic Medical Task Relationships: A Contrastive Learning Perspectivecited by 0
  10. 2026DeCoVec: Building Decoding Space based Task Vector for Large Language Models via In-Context Learningcited by 0
  11. 2026When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformerscited by 0
  12. 2026Prior Directions: Why GUI Grounding Gets Locked in the Pastcited by 0