mlsanityData to 7 Oct 2026

Fast & Faithful Function Vectors

arXiv 2606.05079 ↗First posted 3 Jun 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Function vectors (FVs) are task representations elicited during in-context learning that can be used to steer Large Language Models (LLMs). However, design choices in their formulation remain underexplored. In this work, we study the impact of varying FV definitions for instructions along two degrees of freedom: attention head selection and steering. For head selection, using gradient-based attributions with Layer-wise Relevance Propagation (LRP) substantially improves efficiency as well as accuracy. For FV steering, applying it in a distributed manner yields a higher accuracy compared to simple aggregation. Our code is publicly available.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 40 of its 47 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

40 of 47 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  6. 2019Language Models as Knowledge Bases?cited by 1,922
  7. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  8. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  9. 2019Are Sixteen Heads Really Better than One?cited by 803
  10. 2022Transformers learn in-context by gradient descentcited by 714
  11. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  12. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
28 more
  1. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  2. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  3. 2017Word Translation Without Parallel Datacited by 480
  4. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  5. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  6. 2023In-Context Learning Creates Task Vectorscited by 302
  7. 2023Function Vectors in Large Language Modelscited by 292
  8. 2013Direct and Indirect Effectscited by 276
  9. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  10. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  11. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  12. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  13. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 116
  14. 2022XAI for Transformers: Better Explanations through Conservative Propagationcited by 100
  15. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  16. 2019The emergence of number and syntax units in LSTM language modelscited by 92
  17. 2025Which Attention Heads Matter for In-Context Learning?cited by 61
  18. 2024Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representationscited by 45
  19. 2018Deep RNNs Encode Soft Hierarchical Syntaxcited by 34
  20. 2024The broader spectrum of in-context learningcited by 33
  21. 2017Distinguishing Antonyms and Synonyms in a Pattern-based Neural Networkcited by 19
  22. 2026Language Model Circuits Are Sparse in the Neuron Basiscited by 18
  23. 2025The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentationcited by 11
  24. 2025A Close Look at Decomposition-based XAI-Methods for Transformer Language Modelscited by 10
  25. 2025Do different prompting methods yield a common task representation in language models?cited by 9
  26. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesiscited by 8
  27. 2026From Weights to Activations: Is Steering the Next Frontier of Adaptation?cited by 8
  28. 2025Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insightcited by 3

Built on by

Not cited yet.