# Formalizing and Mitigating Structural Distortion in LLM Attention for Graph Reasoning

arXiv: https://arxiv.org/abs/2606.15633 · HTML: https://mlsanity.com/paper/2606.15633/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 8 of its 60 references that are in the map.
- First posted on arXiv 14 Jun 2026.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 8 of 8)

- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [GraphText: Graph Reasoning in Text Space](https://mlsanity.com/paper/2310.01089/) (2023)
- [Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learning](https://mlsanity.com/paper/2410.07074/) (2024)
- [Exploring Graph Learning Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation](https://mlsanity.com/paper/2502.18771/) (2025)
- [Lost-in-Distance: Impact of Contextual Proximity on LLM Performance in Graph Tasks](https://mlsanity.com/paper/2410.01985/) (2024)
- [AI Benchmarks and Datasets for LLM Evaluation](https://mlsanity.com/paper/2412.01020/) (2024)
- [Unleashing the Potential of Text-attributed Graphs: Automatic Relation Decomposition via Large Language Models](https://mlsanity.com/paper/2405.18581/) (2024)
- [LLaTA: Unlocking Graph Structure Learning with Tree-Guided Large Language Models](https://mlsanity.com/paper/2503.21223/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have shown promise for reasoning over Text-Attributed Graphs (TAGs). However, applying LLMs to graphs requires linearizing their structure into sequences, introducing distortion rooted in the graph bandwidth problem. While this distortion has been shown to degrade performance, it is often attributed to prompt design or model scale, leaving the underlying mechanism unclear. In this work, we show textithow rotary positional embeddings turn graph linearization into bandwidth-dependent attention decay, suppressing attention between graph-adjacent nodes that are forced far apart in the serialized sequence. This shifts the focus of LLM-based graph reasoning from prompt engineering and scaling toward correcting attention misalignment. Motivated by this analysis, we propose Graph-aligned Language Attention (GaLA), a lightweight, inference-time modification for LLMs. GaLA biases attention toward graph-adjacent nodes while preserving the LLM's sequential inductive biases. Across TAG benchmarks, GaLA improves performance with negligible overhead, demonstrating that distortion is a correctable bottleneck in LLM-based graph reasoning.