# GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphs

arXiv: https://arxiv.org/abs/2310.15109 · HTML: https://mlsanity.com/paper/2310.15109/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [Training Graph Neural Networks with 1000 Layers](https://mlsanity.com/paper/2106.07476/) (2021)
- [Learning on Large-scale Text-attributed Graphs via Variational Inference](https://mlsanity.com/paper/2210.14709/) (2022)
- [GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph](https://mlsanity.com/paper/2105.02605/) (2021)
- [Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction](https://mlsanity.com/paper/2111.00064/) (2021)
- [NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphs](https://mlsanity.com/paper/2206.04910/) (2022)
- [Augmentation-Free Self-Supervised Learning on Graphs](https://mlsanity.com/paper/2112.02472/) (2021)
- [Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings](https://mlsanity.com/paper/2202.06671/) (2022)
- [TextGNN: Improving Text Encoder via Graph Neural Network in Sponsored Search](https://mlsanity.com/paper/2101.06323/) (2021)
- [Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphs](https://mlsanity.com/paper/2304.10668/) (2023)
- [Minimally-Supervised Structure-Rich Text Categorization via Learning on Text-Rich Networks](https://mlsanity.com/paper/2102.11479/) (2021)

## Built on by (most cited first, 29 of 29)

- [LightRAG: Simple and Fast Retrieval-Augmented Generation](https://mlsanity.com/paper/2410.05779/) (2024)
- [Large Language Models on Graphs: A Comprehensive Survey](https://mlsanity.com/paper/2312.02783/) (2023)
- [A Survey of Graph Meets Large Language Model: Progress and Future Directions](https://mlsanity.com/paper/2311.12399/) (2023)
- [A Survey of Large Language Models for Graphs](https://mlsanity.com/paper/2405.08011/) (2024)
- [Can LLM Graph Reasoning Generalize beyond Pattern Memorization?](https://mlsanity.com/paper/2406.15992/) (2024)
- [Heterogeneous Contrastive Learning for Foundation Models and Beyond](https://mlsanity.com/paper/2404.00225/) (2024)
- [LLM as GNN: Graph Vocabulary Learning for Text-Attributed Graph Foundation Models](https://mlsanity.com/paper/2503.03313/) (2025)
- [Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learning](https://mlsanity.com/paper/2410.07074/) (2024)
- [When Do LLMs Help With Node Classification? A Comprehensive Analysis](https://mlsanity.com/paper/2502.00829/) (2025)
- [Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision?](https://mlsanity.com/paper/2412.08174/) (2024)
- [A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspective](https://mlsanity.com/paper/2403.16137/) (2024)
- [Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applications](https://mlsanity.com/paper/2510.21131/) (2025)
- [Efficient End-to-end Language Model Fine-tuning on Graphs](https://mlsanity.com/paper/2312.04737/) (2023)
- [TAGA: Text-Attributed Graph Self-Supervised Learning by Synergizing Graph and Text Mutual Transformations](https://mlsanity.com/paper/2405.16800/) (2024)
- [Multi-Scale Heterogeneous Text-Attributed Graph Datasets From Diverse Domains](https://mlsanity.com/paper/2412.08937/) (2024)
- [From Anchors to Answers: A Novel Node Tokenizer for Integrating Graph Structure into Large Language Models](https://mlsanity.com/paper/2410.10743/) (2024)
- [Integrating Structural and Semantic Signals in Text-Attributed Graphs with BiGTex](https://mlsanity.com/paper/2504.12474/) (2025)
- [GNN-as-Judge: Unleashing the Power of LLMs for Graph Learning with GNN Feedback](https://mlsanity.com/paper/2604.08553/) (2026)
- [Large Language Models Meet Graph Neural Networks: A Perspective of Graph Mining](https://mlsanity.com/paper/2412.19211/) (2024)
- [Learning Noise-Resilient and Transferable Graph-Text Alignment via Dynamic Quality Assessment](https://mlsanity.com/paper/2510.19384/) (2025)
- [Path-LLM: A Shortest-Path-based LLM Learning for Unified Graph Representation](https://mlsanity.com/paper/2408.05456/) (2024)
- [Bridging Thoughts and Words: Graph-Based Intent-Semantic Joint Learning for Fake News Detection](https://mlsanity.com/paper/2509.01660/) (2025)
- [DualResearch: Entropy-Gated Dual-Graph Retrieval for Answer Reconstruction](https://mlsanity.com/paper/2510.08959/) (2025)
- [Learning Hierarchical Knowledge in Text-Rich Networks with Taxonomy-Informed Representation Learning](https://mlsanity.com/paper/2603.08159/) (2026)
- [GeoGNN: Quantifying and Mitigating Semantic Drift in Text-Attributed Graphs](https://mlsanity.com/paper/2511.09042/) (2025)
- [S2Aligner: Pair-Efficient and Transferable Pre-Training for Sparse Text-Attributed Graphs](https://mlsanity.com/paper/2605.18579/) (2026)
- [Beyond the Golden Teacher: Enhancing Graph Learning through LLM-GNN Co-teaching](https://mlsanity.com/paper/2606.11583/) (2026)
- [GraspLLM: Towards Zero-Shot Generalization on Text-Attributed Graphs with LLMs](https://mlsanity.com/paper/2606.11898/) (2026)
- [TaLK: Text-attributed Graph Dataset Distillation via Coupling Language Model with Graph-Aware Kernel](https://mlsanity.com/paper/2606.22975/) (2026)

## Abstract (arXiv)

Self-supervised representation learning on text-attributed graphs, which aims to create expressive and generalizable representations for various downstream tasks, has received increasing research attention lately. However, existing methods either struggle to capture the full extent of structural context information or rely on task-specific training labels, which largely hampers their effectiveness and generalizability in practice. To solve the problem of self-supervised representation learning on text-attributed graphs, we develop a novel Graph-Centric Language model -- GRENADE. Specifically, GRENADE exploits the synergistic effect of both pre-trained language model and graph neural network by optimizing with two specialized self-supervised learning algorithms: graph-centric contrastive learning and graph-centric knowledge alignment. The proposed graph-centric self-supervised learning algorithms effectively help GRENADE to capture informative textual semantics as well as structural context information on text-attributed graphs. Through extensive experiments, GRENADE shows its superiority over state-of-the-art methods. Implementation is available at urlhttps://github.com/bigheiniu/GRENADE.