mlsanityData to 14 Oct 2026

Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications

arXiv 2306.02592 ↗First posted 5 Jun 2023cs.CL cs.AI cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Model pre-training on large text corpora has been demonstrated effective for various downstream applications in the NLP domain. In the graph mining domain, a similar analogy can be drawn for pre-training graph models on large graphs in the hope of benefiting downstream graph applications, which has also been explored by several recent studies. However, no existing study has ever investigated the pre-training of text plus graph models on large heterogeneous graphs with abundant textual information (a.k.a. large graph corpora) and then fine-tuning the model on different related downstream applications with different graph schemas. To address this problem, we propose a framework of graph-aware language model pre-training (GALM) on a large graph corpus, which incorporates large language models and graph neural networks, and a variety of fine-tuning methods on downstream applications. We conduct extensive experiments on Amazon's real internal datasets and large public datasets. Comprehensive empirical results and in-depth analysis demonstrate the effectiveness of our proposed methods along with lessons learned.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 34 of its 44 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

34 of 44 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  6. 2017Inductive Representation Learning on Large Graphscited by 6,320
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2018How Powerful are Graph Neural Networks?cited by 4,393
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  11. 2019Strategies for Pre-training Graph Neural Networkscited by 971
  12. 2019Heterogeneous Graph Attention Networkcited by 734
22 more
  1. 2020Heterogeneous Graph Transformercited by 582
  2. 2020GCC: Graph Contrastive Coding for Graph Neural Network Pre-Trainingcited by 463
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  4. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  5. 2018Graph Convolutional Networks for Text Classificationcited by 345
  6. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 303
  7. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  8. 2021Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Predictioncited by 138
  9. 2020Transfer Learning of Graph Neural Networks with Ego-graph Information Maximizationcited by 93
  10. 2019Relational Graph Attention Networkscited by 82
  11. 2020JAKET: Joint Pre-training of Knowledge Graph and Language Understandingcited by 73
  12. 2022Does GNN Pretraining Help Molecular Representation?cited by 60
  13. 2021JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphscited by 58
  14. 2021TextGNN: Improving Text Encoder via Graph Neural Network in Sponsored Searchcited by 54
  15. 2020Knowledge-Aware Language Model Pretrainingcited by 52
  16. 2020Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphscited by 38
  17. 2021AdsGNN: Behavior-Graph Augmented Relevance Modeling in Sponsored Searchcited by 38
  18. 2021Adaptive Transfer Learning on Graph Neural Networkscited by 31
  19. 2021GNN-LM: Language Modeling based on Global Contexts via GNNcited by 27
  20. 2020Exploiting Structured Knowledge in Text via Graph-Guided Representation Learningcited by 26
  21. 2022Efficient and effective training of language and graph neural network modelscited by 9
  22. 2020Pre-Trained Models for Heterogeneous Information Networkscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 16; 2025: 10; 2026: 3.

33 papers.

Sort:

By area:

  1. 2024LightRAG: Simple and Fast Retrieval-Augmented Generationcited by 315
  2. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 192
  3. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 103
  4. 2024Position: Graph Foundation Models are Already Herecited by 86
  5. 2023A Survey of Graph Meets Large Language Model: Progress and Future Directionscited by 85
  6. 2024A Survey of Large Language Models for Graphscited by 72
  7. 2024GraphTranslator: Aligning Graph Model to Large Language Model for Open-ended Taskscited by 63
  8. 2024Advancing Graph Representation Learning with Large Language Models: A Comprehensive Survey of Techniquescited by 16
  9. 2023TouchUp-G: Improving Feature Representation through Graph-Centric Finetuningcited by 12
  10. 2024Towards Graph Foundation Models for Personalizationcited by 12
  11. 2024Exploring the Potential of Large Language Models for Heterophilic Graphscited by 10
  12. 2024A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspectivecited by 9
21 more
  1. 2024Lost-in-Distance: Impact of Contextual Proximity on LLM Performance in Graph Taskscited by 8
  2. 2024GraphStorm: all-in-one graph machine learning framework for industry applicationscited by 7
  3. 2024A Survey of Few-Shot Learning on Graphs: from Meta-Learning to Pre-Training and Prompt Learningcited by 6
  4. 2025Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applicationscited by 6
  5. 2024Enhancing Collaborative Semantics of Language Model-Driven Recommendations via Graph-Aware Learningcited by 5
  6. 2025HeTGB: A Comprehensive Benchmark for Heterophilic Text-Attributed Graphscited by 5
  7. 2024IOHunter: Graph Foundation Model to Uncover Online Information Operationscited by 4
  8. 2024Large Language Models Meet Graph Neural Networks: A Perspective of Graph Miningcited by 3
  9. 2025Data-centric Federated Graph Learning with Large Language Modelscited by 3
  10. 2025Unify Graph Learning with Text: Unleashing LLM Potentials for Session Searchcited by 3
  11. 2025Dynamic Bundling with Large Language Models for Zero-Shot Inference on Text-Attributed Graphscited by 3
  12. 2025Adversarial Attacks and Defenses on Graph-aware Large Language Models (LLMs)cited by 3
  13. 2025A Scalable and Efficient Signal Integration System for Job Matchingcited by 2
  14. 2024SemPool: Simple, robust, and interpretable KG pooling for enhancing language modelscited by 1
  15. 2024Privately Learning from Graphs with Applications in Fine-tuning Large Language Modelscited by 1
  16. 2025DualResearch: Entropy-Gated Dual-Graph Retrieval for Answer Reconstructioncited by 1
  17. 2026MambaFormer: Token-Level Guided Routing Mixture-of-Experts for Accurate and Efficient Clinical Assistancecited by 1
  18. 2026Node Role-Guided LLMs for Dynamic Graph Clusteringcited by 1
  19. 2026Let Relations Speak: An End-to-End LLM-GNN Soft Prompt Framework for Fraud Detectioncited by 1
  20. 2025Differentially Private Relational Learning with Entity-level Privacy Guaranteescited by 0
  21. 2025Enhancing Spectral Graph Neural Networks with LLM-Predicted Homophilycited by 0