mlsanityData to 14 Oct 2026

RIFT: Token Influence Follows the Path of Least Resistance

arXiv 2610.16349 ↗First posted 12 Oct 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Which context tokens drive a language model's answer? Graph-based tools such as attention rollout and attention flow answer this from a single forward pass, yet their faithfulness in decoder-only models is rarely tested against interventions. We view the layers of a decoder as a causal network whose conductances are attention weights, and read a token's influence on the answer from the resistance between them. We propose RIFT (Resistance-based Information Flow in Transformers), which scores a token by the series resistance of its path of least resistance to the answer, found exactly by a classic shortest-path dynamic program. On synthetic in-context retrieval tasks and extractive question answering over Wikipedia paragraphs, RIFT predicts the effect of replacing a token with a Spearman correlation of 0.52-0.68; predicting a single replacement, it reaches 77-96% of the agreement between two different random replacements. It needs no gradients and no extra forward passes, scoring every token of a prompt in 0.9-3.4 ms on one CPU thread, whereas the replacement experiment needs a forward pass per token. On 576 held-out prompts from four models of two families (Qwen3 and Llama 3.2), it outperforms token distance, attention rollout and attention flow in every pre-specified comparison, by 0.23 to 0.88 in mean Spearman correlation, and it is the best or tied-best of all attention-based scores we test. Aggregates that add up all routes (attention flow, effective resistance and rollout) instead mostly encode token position: as a simple proposition shows, early tokens have many more parallel routes to the answer.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 37 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

33 of 37 references.

References · 33

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2022In-context Learning and Induction Headscited by 964
  7. 2019Attention is not Explanationcited by 929
  8. 2020Quantifying Attention Flow in Transformerscited by 775
  9. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  10. 2019Attention is not not Explanationcited by 636
  11. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  12. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
21 more
  1. 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 478
  2. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  3. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  4. 2023On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topologycited by 166
  5. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 120
  6. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 111
  7. 2024ContextCite: Attributing Model Generation to Contextcited by 98
  8. 2023Understanding Oversquashing in GNNs through the Lens of Effective Resistancecited by 97
  9. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 95
  10. 2024Transformers need glasses! Information over-squashing in language taskscited by 91
  11. 2022Measuring the Mixing of Contextual Information in the Transformercited by 65
  12. 2025On the Emergence of Position Bias in Transformerscited by 59
  13. 2025Learning to Attribute with Attentioncited by 13
  14. 2025Attention Consistency for LLMs Explanationcited by 11
  15. 2026A Structural Theory of Position Bias in Transformerscited by 5
  16. 2025Generalized Attention Flow: Feature Attribution for Transformer Models via Maximum Flowcited by 4
  17. 2026Lost in the Middle at Birth: An Exact Theory of Transformer Position Biascited by 4
  18. 2023Multi-Layer Attention-Based Explainability via Transformers for Tabular Datacited by 2
  19. 2026Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMscited by 2
  20. 2022Attention Flows for General Transformerscited by 1
  21. 2025Understanding the Failure Modes of Transformers through the Lens of Graph Neural Networkscited by 1

Built on by

Not cited yet.