mlsanityData to 7 Oct 2026

What Structural Inductive Bias Helps Transformers Reason Over Knowledge Graphs? A Study with Tabula RASA

arXiv 2602.02834 ↗First posted 2 Feb 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

What structural inductive bias helps transformers reason over knowledge graphs? Through controlled ablations of a minimal transformer modification with four independently removable components (sparse adjacency masking, edge-type biases, query scaling, value gating), we isolate which structural signals drive multi-hop reasoning. Our finding is sharp: sparse adjacency masking alone accounts for the dominant share of improvement over unmasked transformers (+72.5pp on 3-hop MetaQA, +45.5pp on WebQSP, +53.9pp on CWQ), while learned relation parameters add only modest refinement and can actively hurt without structural guidance. A zero-shot experiment provides architecturally independent corroboration: masking-based attention degrades 4.0x less than relation-specific weights when edge types are held out. The useful inductive bias for multi-hop KGQA is predominantly topological, not relational.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

26 of 32 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Graph Attention Networkscited by 8,854
  3. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  4. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  5. 2021How Attentive are Graph Attention Networks?cited by 818
  6. 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
  7. 2020A Generalization of Transformer Networks to Graphscited by 653
  8. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  9. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  10. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  11. 2021QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answeringcited by 386
  12. 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 326
14 more
  1. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 318
  2. 2022Structure-Aware Transformer for Graph Representation Learningcited by 191
  3. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 191
  4. 2017Variational Reasoning for Question Answering with Knowledge Graphcited by 170
  5. 2020Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddingscited by 169
  6. 2023Exphormer: Sparse Transformers for Graphscited by 166
  7. 2023Graph Inductive Biases in Transformers without Message Passingcited by 150
  8. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 137
  9. 2023Representational Strengths and Limitations of Transformerscited by 132
  10. 2024GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoningcited by 128
  11. 2021Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signalscited by 100
  12. 2023Where Did the Gap Go? Reassessing the Long-Range Graph Benchmarkcited by 83
  13. 2024Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generationcited by 78
  14. 2022ReaRev: Adaptive Reasoning for Question Answering over Knowledge Graphscited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Transformers Can Learn Connectivity in Some Graphs but Not Others
  2. 2025How do Transformers Learn Implicit Reasoning?
  3. 2025Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge
  4. 2026Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
  5. 2026Structure-Internalized Rule Language Model for Faithful Knowledge Graph Reasoning
  6. 2025From Sequence to Structure: Uncovering Substructure Reasoning in Transformers
  7. 2025Uncovering Graph Reasoning in Decoder-only Transformers with Circuit Tracing
  8. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization