mlsanityData to 14 Oct 2026

Patton: Language Model Pretraining on Text-Rich Networks

arXiv 2305.12268 ↗First posted 20 May 2023cs.CL cs.AI cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

A real-world text corpus sometimes comprises not only text documents but also semantic links between them (e.g., academic papers in a bibliographic network are linked by citations and co-authorships). Text documents and semantic connections form a text-rich network, which empowers a wide range of downstream tasks such as classification and retrieval. However, pretraining methods for such structures are still lacking, making it difficult to build one generic model that can be adapted to various tasks on text-rich networks. Current pretraining objectives, such as masked language modeling, purely model texts and do not take inter-document structure information into consideration. To this end, we propose our PretrAining on TexT-Rich NetwOrk framework Patton. Patton includes two pretraining strategies: network-contextualized masked language modeling and masked node prediction, to capture the inherent dependency between textual attributes and network structure. We conduct experiments on four downstream tasks in five datasets from both academic and e-commerce domains, where Patton outperforms baselines significantly and consistently.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 20 May 2023.

Builds on

29 of 45 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  8. 2018Deep contextualized word representationscited by 2,290
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  11. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  12. 2018Deep Graph Infomaxcited by 1,106
17 more
  1. 2019Strategies for Pre-training Graph Neural Networkscited by 971
  2. 2019Heterogeneous Graph Attention Networkcited by 734
  3. 2015Image-based Recommendations on Styles and Substitutescited by 706
  4. 2020GCC: Graph Contrastive Coding for Graph Neural Network Pre-Trainingcited by 463
  5. 2019InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximizationcited by 441
  6. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
  7. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
  8. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 303
  9. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  10. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 211
  11. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 144
  12. 2021GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graphcited by 143
  13. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  14. 2022Heterformer: Transformer-based Deep Node Representation Learning on Heterogeneous Text-Rich Networkscited by 24
  15. 2023Edgeformers: Graph-Empowered Transformers for Representation Learning on Textual-Edge Networkscited by 23
  16. 2020Large Scale Subject Category Classification of Scholarly Papers with Deep Attentive Neural Networkscited by 9
  17. 2014Exploiting citation networks for large-scale author name disambiguationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 24; 2025: 11; 2026: 5.

49 papers.

Sort:

By area:

  1. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 192
  2. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 176
  3. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 103
  4. 2024Position: Graph Foundation Models are Already Herecited by 86
  5. 2023A Survey of Graph Meets Large Language Model: Progress and Future Directionscited by 85
  6. 2024Can GNN be Good Adapter for LLMs?cited by 71
  7. 2023Graph Prompt Learning: A Comprehensive Survey and Beyondcited by 61
  8. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 36
  9. 2024GLBench: A Comprehensive Benchmark for Graph with Large Language Modelscited by 36
  10. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  11. 2025When Do LLMs Help With Node Classification? A Comprehensive Analysiscited by 21
  12. 2024Advancing Graph Representation Learning with Large Language Models: A Comprehensive Survey of Techniquescited by 16
37 more
  1. 2024HLM-Cite: Hybrid Language Model Workflow for Text-based Scientific Citation Predictioncited by 15
  2. 2024UniGLM: Training One Unified Language Model for Text-Attributed Graph Embeddingcited by 14
  3. 2024InstructG2I: Synthesizing Images from Multimodal Attributed Graphscited by 14
  4. 2024Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision?cited by 13
  5. 2023TouchUp-G: Improving Feature Representation through Graph-Centric Finetuningcited by 12
  6. 2025GTool: Graph Enhanced Tool Planning with Large Language Modelcited by 10
  7. 2024A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspectivecited by 9
  8. 2024LinkGPT: Teaching Large Language Models To Predict Missing Linkscited by 9
  9. 2024TEG-DB: A Comprehensive Dataset and Benchmark of Textual-Edge Graphscited by 9
  10. 2024Large Language Model Enhanced Knowledge Representation Learning: A Surveycited by 7
  11. 2023When Large Language Models Meet Citation: A Surveycited by 6
  12. 2024A Survey of Few-Shot Learning on Graphs: from Meta-Learning to Pre-Training and Prompt Learningcited by 6
  13. 2025Unifying Text Semantics and Graph Structures for Temporal Text-attributed Graphs with Large Language Modelscited by 6
  14. 2025Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applicationscited by 6
  15. 2023Efficient End-to-end Language Model Fine-tuning on Graphscited by 5
  16. 2024TAGA: Text-Attributed Graph Self-Supervised Learning by Synergizing Graph and Text Mutual Transformationscited by 5
  17. 2024Investigating Instruction Tuning Large Language Models on Graphscited by 5
  18. 2023Weakly Supervised Multi-Label Classification of Full-Text Scientific Paperscited by 4
  19. 2024Parameter-Efficient Tuning Large Language Models for Graph Representation Learningcited by 3
  20. 2024Hierarchical Compression of Text-Rich Graphs via Large Language Modelscited by 3
  21. 2024Bridging Large Language Models and Graph Structure Learning Models for Robust Representation Learningcited by 3
  22. 2024The Graph's Apprentice: Teaching an LLM Low Level Knowledge for Circuit Quality Estimationcited by 3
  23. 2024Large Language Models Meet Graph Neural Networks: A Perspective of Graph Miningcited by 3
  24. 2025Learning Noise-Resilient and Transferable Graph-Text Alignment via Dynamic Quality Assessmentcited by 3
  25. 2025HierPromptLM: A Pure PLM-based Framework for Representation Learning on Heterogeneous Text-rich Networkscited by 2
  26. 2025A Graph Talks, But Who's Listening? Rethinking Evaluations for Graph-Language Modelscited by 2
  27. 2026GraphScout: Empowering Large Language Models with Intrinsic Exploration Ability for Agentic Graph Reasoningcited by 2
  28. 2024Towards Versatile Graph Learning Approach: from the Perspective of Large Language Modelscited by 1
  29. 2024Privately Learning from Graphs with Applications in Fine-tuning Large Language Modelscited by 1
  30. 2025Adapting Pretrained Language Models for Citation Classification via Self-Supervised Contrastive Learningcited by 1
  31. 2025CITE: A Comprehensive Benchmark for Heterogeneous Text-Attributed Graphs on Catalytic Materialscited by 1
  32. 2026Learning Hierarchical Knowledge in Text-Rich Networks with Taxonomy-Informed Representation Learningcited by 1
  33. 2026ERAlign: Energy-based Representation Alignment of GNNs and LLMs on Text-attributed Graphscited by 1
  34. 2025Hierarchical Graph Topic Modeling with Topic Tree-based Transformercited by 0
  35. 2025Odin: Oriented Dual-module Integration for Text-rich Network Representation Learningcited by 0
  36. 2026Scaling GraphLLM with Bilevel-Optimized Sparse Queryingcited by 0
  37. 2026OOD-GraphLLM: Graph Large Language Model for Out-of-Distribution Generalized Drug Synergy Predictioncited by 0