mlsanityData to 7 Oct 2026

Graph Language Models

arXiv 2401.07105 ↗First posted 13 Jan 2024cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

While Language Models (LMs) are the workhorses of NLP, their interplay with structured knowledge graphs (KGs) is still actively researched. Current methods for encoding such graphs typically either (i) linearize them for embedding with LMs -- which underutilize structural information, or (ii) use Graph Neural Networks (GNNs) to preserve the graph structure -- but GNNs cannot represent text features as well as pretrained LMs. In our work we introduce a novel LM type, the Graph Language Model (GLM), that integrates the strengths of both approaches and mitigates their weaknesses. The GLM parameters are initialized from a pretrained LM to enhance understanding of individual graph concepts and triplets. Simultaneously, we design the GLM's architecture to incorporate graph biases, thereby promoting effective knowledge distribution within the graph. This enables GLMs to process graphs, texts, and interleaved inputs of both. Empirical evaluations on relation classification tasks show that GLM embeddings surpass both LM- and GNN-based baselines in supervised and zero-shot setting, demonstrating their versatility.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 13 Jan 2024.

Builds on

34 of 49 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2017Graph Attention Networkscited by 8,854
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  7. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  8. 2018Self-Attention with Relative Position Representationscited by 1,112
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  11. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  12. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
22 more
  1. 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 544
  2. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  3. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  4. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  5. 2019KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoningcited by 216
  6. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  7. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  8. 2022Learning on Large-scale Text-attributed Graphs via Variational Inferencecited by 153
  9. 2019Text Generation from Knowledge Graphs with Graph Transformerscited by 122
  10. 2020Investigating Pretrained Language Models for Graph-to-Text Generationcited by 117
  11. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  12. 2022Transformer for Graphs: An Overview from Architecture Perspectivecited by 107
  13. 2021Position Information in Transformers: An Overviewcited by 103
  14. 2019Commonsense Knowledge Base Completion with Structural and Semantic Contextcited by 61
  15. 2016A Convolutional Encoder Model for Neural Machine Translationcited by 54
  16. 2020Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answeringcited by 44
  17. 2021Dict-BERT: Enhancing Language Model Pre-training with Dictionarycited by 38
  18. 2020Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphscited by 30
  19. 2021Few-shot Knowledge Graph-to-Text Generation with Pretrained Language Modelscited by 29
  20. 2024Unifying Structured Data as Graph for Data-to-Text Pre-Trainingcited by 13
  21. 2023NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillationcited by 8
  22. 2023Similarity-weighted Construction of Contextualized Commonsense Knowledge Graphs for Knowledge-intense Argumentation Taskscited by 6

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 4.

6 papers.

  1. 2024GOFA: A Generative One-For-All Model for Joint Graph Language Modelingcited by 56
  2. 2025Set-LLM: A Permutation-Invariant LLMcited by 10
  3. 2024MEG: Medical Knowledge-Augmented Large Language Models for Question Answeringcited by 6
  4. 2025GLaMoR: Consistency Checking of OWL Ontologies using Graph Language Modelscited by 2
  5. 2025GILT: An LLM-Free, Tuning-Free Graph Foundational Model for In-Context Learningcited by 2
  6. 2025GMLM: Bridging Graph Neural Networks and Language Models for Heterophilic Node Classificationcited by 1