mlsanityData to 7 Oct 2026

Spectral Attention Steering for Prompt Highlighting

arXiv 2603.01281 ↗First posted 1 Mar 2026cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Attention steering is an important technique for controlling model focus, enabling capabilities such as prompt highlighting, where the model prioritises user-specified text. However, existing attention steering methods require explicit storage of the full attention matrix, making them incompatible with memory-efficient implementations like FlashAttention. We introduce Spectral Editing Key Amplification (SEKA), a training-free steering method that tackles this by directly editing key embeddings before attention computation. SEKA uses spectral decomposition to steer key embeddings towards latent directions that amplify attention scores for certain tokens. We extend this to Adaptive SEKA (AdaSEKA), a query-adaptive variant that uses a training-free routing mechanism to dynamically combine multiple expert subspaces based on the prompt's semantic intent. Our experiments show both methods significantly outperform strong baselines on standard steering benchmarks while adding much lower latency and memory overhead, in compatibility with optimised attention.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 35 of its 40 references that are in the map. First posted on arXiv 1 Mar 2026.

Builds on

35 of 40 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  10. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  11. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
23 more
  1. 2023Steering Language Models With Activation Engineeringcited by 824
  2. 2019Are Sixteen Heads Really Better than One?cited by 803
  3. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  4. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  5. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  6. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  7. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  8. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  9. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
  10. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  11. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  12. 2024Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilizationcited by 97
  13. 2024Transformers need glasses! Information over-squashing in language taskscited by 87
  14. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
  15. 2024Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categoriescited by 74
  16. 2024Spectral Editing of Activations for Large Language Model Alignmentcited by 54
  17. 2025Unlocking General Long Chain-of-Thought Reasoning Capabilities of Large Language Models via Representation Engineeringcited by 34
  18. 2023Shielded Representations: Protecting Sensitive Attributes Through Iterative Gradient-Based Projectioncited by 25
  19. 2024Towards Inference-time Category-wise Safety Steering for Large Language Modelscited by 17
  20. 2024Selective Prompt Anchoring for Code Generationcited by 16
  21. 2024Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactionscited by 11
  22. 2023BERT is not The Count: Learning to Match Mathematical Statements with Proofscited by 9
  23. 2025HACK: Hallucinations Along Certainty and Knowledge Axescited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026PRISM-Delta: Differential Subspace Steering for Prompt Highlighting in Large Language Modelscited by 2
  2. 2026Summoning the Oracle to Slay It: Mitigating Look-Ahead Bias in Financial Backtesting with Large Language Modelscited by 2
  3. 2026Memory Inception: Latent-Space KV Cache Manipulation for Steering LLMscited by 1
  4. 2026Enhancing Multi-Agent Communication through Attention Steering with Context Relevancecited by 0
  5. 2026Attention is Case-Sensitivecited by 0
  6. 2026When Can Attention Heads Be Statically Defined?cited by 0