mlsanityData to 14 Oct 2026

Transformers as Graph-to-Graph Models

arXiv 2310.17936 ↗First posted 27 Oct 2023cs.CL cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We argue that Transformers are essentially graph-to-graph models, with sequences just being a special case. Attention weights are functionally equivalent to graph edges. Our Graph-to-Graph Transformer architecture makes this ability explicit, by inputting graph edges into the attention weight computations and predicting graph edges with attention-like functions, thereby integrating explicit graphs into the latent graphs learned by pretrained Transformers. Adding iterative graph refinement provides a joint embedding of input, output, and latent graphs, allowing non-autoregressive graph prediction to optimise the complete graph without any bespoke pipeline or decoding strategy. Empirical results show that this architecture achieves state-of-the-art accuracies for modelling a variety of linguistic structures, integrating very effectively with the latent linguistic representations learned by pretraining.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 60 of its 96 references that are in the map. First posted on arXiv 27 Oct 2023.

Builds on

60 of 96 references.

References · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  5. 2017Graph Attention Networkscited by 8,923
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  9. 2018Deep contextualized word representationscited by 2,290
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2020Rethinking Attention with Performerscited by 1,601
  12. 2018Self-Attention with Relative Position Representationscited by 1,120
38 more
  1. 2013Spectral Networks and Locally Connected Networks on Graphscited by 1,102
  2. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  3. 2015Gated Graph Sequence Neural Networkscited by 796
  4. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 631
  5. 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 478
  6. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 320
  7. 2017End-to-end Neural Coreference Resolutioncited by 241
  8. 2018Higher-order Coreference Resolution with Coarse-to-fine Inferencecited by 150
  9. 2018Graph Convolution over Pruned Dependency Trees Improves Relation Extractioncited by 146
  10. 2017Encoding Sentences with Graph Convolutional Networks for Semantic Role Labelingcited by 137
  11. 2019BERT for Coreference Resolution: Baselines and Analysiscited by 125
  12. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  13. 2021Position Information in Transformers: An Overviewcited by 103
  14. 2022TableFormer: Robust Transformer Modeling for Table-Text Encodingcited by 91
  15. 201975 Languages, 1 Model: Parsing Universal Dependencies Universallycited by 90
  16. 2020Improve Transformer Models with Better Relative Position Embeddingscited by 73
  17. 2018Linguistically-Informed Self-Attention for Semantic Role Labelingcited by 71
  18. 2021Entity Structure Within and Throughout: Modeling Mention Dependencies for Document-Level Relation Extractioncited by 68
  19. 2021Conditional probing: measuring usable information beyond a baselinecited by 62
  20. 2021JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphscited by 58
  21. 2016Improving Coreference Resolution by Learning Entity-Level Distributed Representationscited by 50
  22. 2020Revealing the Myth of Higher-Order Inference in Coreference Resolutioncited by 49
  23. 2015Transition-Based Dependency Parsing with Stack Long Short-Term Memorycited by 45
  24. 2019Head-Driven Phrase Structure Grammar Parsing on Penn Treebankcited by 43
  25. 2022Relational Attention: Generalizing Transformers for Graph-Structured Taskscited by 41
  26. 2021Relative Positional Encoding for Transformers with Linear Complexitycited by 37
  27. 2020Asking without Telling: Exploring Latent Ontologies in Contextual Representationscited by 34
  28. 2016Globally Normalized Transition-Based Neural Networkscited by 26
  29. 2016Learning Global Features for Coreference Resolutioncited by 20
  30. 2020Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinementcited by 20
  31. 2017Improved Neural Machine Translation with a Syntax-Aware Encoder and Decodercited by 16
  32. 2021PermuteFormer: Efficient Relative Position Encoding for Long Sequencescited by 14
  33. 2019Parsing All: Syntax and Semantics, Dependencies and Spanscited by 13
  34. 2019Graph-to-Graph Transformer for Transition-based Dependency Parsingcited by 13
  35. 2019Graph Convolutions over Constituent Trees for Syntax-Aware Semantic Role Labelingcited by 12
  36. 2020The Unstoppable Rise of Computational Linguistics in Deep Learningcited by 12
  37. 2015Structured Training for Neural Network Transition-Based Parsingcited by 11
  38. 2016Distilling an Ensemble of Greedy Dependency Parsers into One MST Parsercited by 11

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 2.

5 papers.

  1. 2024GOAL: A Generalist Combinatorial Optimization Agent Learnercited by 42
  2. 2024Quality Diversity for Robot Learning: Limitations and Future Directionscited by 1
  3. 2025GMLM: Bridging Graph Neural Networks and Language Models for Heterophilic Node Classificationcited by 1
  4. 2025Understanding the Staged Dynamics of Transformers in Learning Latent Structurecited by 1
  5. 2024Fast-and-Frugal Text-Graph Transformers are Effective Link Predictorscited by 0