mlsanityData to 14 Oct 2026

Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relay

arXiv 2610.05564 ↗First posted 4 Oct 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Text embedding models trained with contrastive learning learn to follow task instructions from instruction-paired data, while instruction-tuned LLMs already know how to follow them. We show that this instruction-following ability can carry over from an LLM to a Transformer-based embedder without any training. We propose Attention Relay, which passes the attention weights an LLM produces to the embedder's own attention. Across six instruction-tuned LLMs from the Qwen3, Llama 3.1 and OLMo 3 families and ten widely used embedding models that differ in tokenizer, size and pooling type, Attention Relay makes nearly every combination instruction-aware. Experiments that break the method down into its parts show that the LLM's attention weights track the instruction in its later layers and come largely from instruction tuning. They also show that relaying these weights selects which content in the text matters: it makes the aspect of the text that the instruction asks about dominant in the embedding, or restores that aspect where averaging had diluted it.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 49 of its 56 references that are in the map. First posted on arXiv 4 Oct 2026.

Builds on

49 of 56 references.

References · 49

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  9. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  10. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  11. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  12. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
37 more
  1. 2019Attention is not Explanationcited by 929
  2. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  3. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  4. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  5. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  6. 2019Attention is not not Explanationcited by 636
  7. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  8. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  9. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  10. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  11. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  12. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 333
  13. 2023Improving Text Embeddings with Large Language Modelscited by 303
  14. 2024Generative Representational Instruction Tuningcited by 233
  15. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  16. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 176
  17. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 101
  18. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 91
  19. 2024The Hidden Attention of Mamba Modelscited by 91
  20. 2023Scaling Sentence Embeddings with Large Language Modelscited by 85
  21. 2024Repetition Improves Language Model Embeddingscited by 72
  22. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 69
  23. 2024Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankerscited by 43
  24. 2025Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-rankingcited by 31
  25. 2024Answer is All You Need: Instruction-following Text Embedding via Answering the Questioncited by 27
  26. 2025Spotlight Your Instructions: Instruction-following with Dynamic Attention Steeringcited by 21
  27. 2024Model Tells Itself Where to Attend: Faithfulness Meets Automatic Attention Steeringcited by 17
  28. 2024On the Surprising Effectiveness of Attention Transfer for Vision Transformerscited by 17
  29. 2024Meta-Task Prompting Elicits Embeddings from Large Language Modelscited by 16
  30. 2024GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddingscited by 15
  31. 2025Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimationcited by 14
  32. 2025Instruction Following by Principled Boosting Attention of Large Language Modelscited by 9
  33. 2025Don't Reinvent the Wheel: Efficient Instruction-Following Text Embedding based on Guided Space Transformationcited by 7
  34. 2025Model-Aware Tokenizer Transfercited by 6
  35. 2025Out-of-the-Box Conditional Text Embeddings from Large Language Modelscited by 5
  36. 2026LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden Statescited by 3
  37. 2026Focusing Condition: Inference-Time Self-Contrastive Steering Elicits Better Conditional Text Embeddings in LLMscited by 1

Built on by

Not cited yet.