# How Token Influence Decays with Distance: A Green-Function View of Trained Language Models

arXiv: https://arxiv.org/abs/2606.29139 · HTML: https://mlsanity.com/paper/2606.29139/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 28 Jun 2026.

## Builds on (most cited first, 19 of 19)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?](https://mlsanity.com/paper/2010.05607/) (2020)
- [Information Flow Routes: Automatically Interpreting Language Models at Scale](https://mlsanity.com/paper/2403.00824/) (2024)
- [Do Long-Range Language Models Actually Use Long-Range Context?](https://mlsanity.com/paper/2109.09115/) (2021)
- [On the Emergence of Position Bias in Transformers](https://mlsanity.com/paper/2502.01951/) (2025)
- [Machine learning and domain decomposition methods -- a survey](https://mlsanity.com/paper/2312.14050/) (2023)
- [A Structural Theory of Position Bias in Transformers](https://mlsanity.com/paper/2602.16837/) (2026)
- [Jacobian Scopes: token-level causal attributions in LLMs](https://mlsanity.com/paper/2601.16407/) (2026)
- [Overlapping Schwarz Attention: Hierarchical Attention via Domain Decomposition](https://mlsanity.com/paper/2606.18525/) (2026)

## Abstract (arXiv)

We study how the next-token prediction of an autoregressive Transformer language model changes under small perturbations of earlier input token embeddings. Motivated by operator learning and iterative solvers for differential equations, we investigate how the influence of one token on another decays with distance in a trained model. In multilevel methods for differential equations, such as domain decomposition, multigrid, and multilevel preconditioning, one often exploits a separation between strong local interactions and weaker but essential global interactions. The latter correspond to the long tail of the Green's function and are typically handled by a coarse-level operator. Inspired by this perspective, we compute an empirical, distance-resolved gradient profile of token dependencies using autograd. Experiments on trained Pythia models and Qwen2.5-0.5B show that, over the measured distance range, the median Jacobian sensitivity is much better described by a power-law-type decay than by an exponential alternative: the diagonal-normalized profile is well described by overline G(r) ≈ γ+β(r+1)-p with exponents p ≈ 0.7--0.9 (typically 0.8--0.9). This behavior appears on coherent text from Gutenberg and WikiText-103. Token-shuffling experiments show that the power-law profile persists even when syntax and prediction quality collapse, whereas randomly initialized models do not exhibit it. The slowly decaying long-range sensitivity thus appears to be a learned property of trained autoregressive Transformer operators. These findings suggest that hierarchical or coarse-level mechanisms in language models may be able to exploit the long-tailed sensitivity profiles.