mlsanityData to 14 Oct 2026

JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs

arXiv 2106.10502 ↗First posted 19 Jun 2021cs.CL cs.AIcited by 58

Not on the map. Why?

Abstract (arXiv)

Existing pre-trained models for knowledge-graph-to-text (KG-to-text) generation simply fine-tune text-to-text pre-trained models such as BART or T5 on KG-to-text datasets, which largely ignore the graph structure during encoding and lack elaborate pre-training tasks to explicitly model graph-text alignments. To tackle these problems, we propose a graph-text joint representation learning model called JointGT. During encoding, we devise a structure-aware semantic aggregation module which is plugged into each Transformer layer to preserve the graph structure. Furthermore, we propose three new pre-training tasks to explicitly enhance the graph-text alignment including respective text / graph reconstruction, and graph-text alignment in the embedding space via Optimal Transport. Experiments show that JointGT obtains new state-of-the-art performance on various KG-to-text datasets.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 32 of its 52 references that are in the map. First posted on arXiv 19 Jun 2021.

Builds on

32 of 52 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  7. 2018Self-Attention with Relative Position Representationscited by 1,120
  8. 2018Computational Optimal Transportcited by 780
  9. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  10. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 410
  11. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  12. 2019Knowledge Enhanced Contextual Word Representationscited by 296
20 more
  1. 2019Text Generation from Knowledge Graphs with Graph Transformerscited by 122
  2. 2020Investigating Pretrained Language Models for Graph-to-Text Generationcited by 117
  3. 2020Text-to-Text Pre-Training for Data-to-Text Taskscited by 104
  4. 2020Graph Optimal Transport for Cross-Domain Alignmentcited by 103
  5. 2019Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generationcited by 75
  6. 2020JAKET: Joint Pre-training of Knowledge Graph and Language Understandingcited by 73
  7. 2018Story Ending Generation with Incremental Encoding and Commonsense Knowledgecited by 60
  8. 2020Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graphcited by 57
  9. 2019Neural data-to-text generation: A comparison between pipeline and end-to-end architecturescited by 51
  10. 2018An Interpretable Reasoning Network for Multi-Relation Question Answeringcited by 47
  11. 2020GPT-too: A language-model-first approach for AMR-to-text generationcited by 38
  12. 2020CycleGT: Unsupervised Graph-to-Text and Text-to-Graph Generation via Cycle Trainingcited by 38
  13. 2019Modeling Graph Structure in Transformer for Better AMR-to-Text Generationcited by 35
  14. 2016Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpuscited by 33
  15. 2020Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphscited by 30
  16. 2018Deep Graph Convolutional Encoders for Structured Data to Text Generationcited by 27
  17. 2020Toward Subgraph-Guided Knowledge Graph Question Generation with Graph Neural Networkscited by 25
  18. 2021Structural Information Preserving for Graph-to-Text Generationcited by 24
  19. 2020Partially-Aligned Data-to-Text Generation with Distant Supervisioncited by 10
  20. 2020Modeling Graph Structure via Relative Position for Text Generation from Knowledge Graphscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 16; 2023: 21; 2024: 10; 2025: 7; 2026: 3.

58 papers.

Sort:

By area:

  1. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  2. 2022Multimodal Learning with Transformers: A Surveycited by 400
  3. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  4. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  5. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  6. 2023Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answeringcited by 65
  7. 2023Graph Neural Prompting with Large Language Modelscited by 64
  8. 2023Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysiscited by 35
  9. 2023Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applicationscited by 33
  10. 2023ConGraT: Self-Supervised Contrastive Pretraining for Joint Graph and Text Embeddingscited by 32
  11. 2022Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methodscited by 27
  12. 2023Using Large Language Models for Zero-Shot Natural Language Generation from Knowledge Graphscited by 24
38 more
  1. 2023Prompting Large Language Models with Chain-of-Thought for Few-Shot Knowledge Base Question Generationcited by 21
  2. 2024Graph-Based Retriever Captures the Long Tail of Biomedical Knowledgecited by 20
  3. 2022Neural Pipeline for Zero-Shot Data-to-Text Generationcited by 19
  4. 2023State2Explanation: Concept-Based Explanations to Benefit Agent Learning and User Understandingcited by 17
  5. 2022GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text Generationcited by 16
  6. 2022MVP: Multi-task Supervised Pre-training for Natural Language Generationcited by 16
  7. 2024Research Trends for the Interplay between Large Language Models and Knowledge Graphscited by 15
  8. 2022Empowering Language Models with Knowledge Graph Reasoning for Question Answeringcited by 14
  9. 2024Unifying Structured Data as Graph for Data-to-Text Pre-Trainingcited by 13
  10. 2024Efficient Knowledge Infusion via KG-LLM Alignmentcited by 13
  11. 2025Graph2text or Graph2token: A Perspective of Large Language Models for Graph Learningcited by 12
  12. 2022Innovations in Neural Data-to-text Generation: A Surveycited by 10
  13. 2023FABULA: Intelligence Report Generation Using Retrieval-Augmented Narrative Constructioncited by 10
  14. 2022AutoQGS: Auto-Prompt for Low-Resource Knowledge-based Question Generation from SPARQLcited by 8
  15. 2022MURMUR: Modular Multi-Step Reasoning for Semi-Structured Data-to-Text Generationcited by 7
  16. 2022Mind the Labels: Describing Relations in Knowledge Graphs With Pretrained Modelscited by 6
  17. 2023Pre-training Language Models for Comparative Reasoningcited by 6
  18. 2023Diversifying Question Generation over Knowledge Base via External Natural Questionscited by 6
  19. 2024SGSH: Stimulate Large Language Models with Skeleton Heuristics for Knowledge Base Question Generationcited by 6
  20. 2025Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applicationscited by 6
  21. 2023Transformers as Graph-to-Graph Modelscited by 5
  22. 2024A Survey on Neural Question Generation: Methods, Applications, and Prospectscited by 5
  23. 2023Generating Faithful Text From a Knowledge Graph with Noisy Reference Textcited by 4
  24. 2023Knowledge-grounded Natural Language Recommendation Explanationcited by 4
  25. 2024A Systematic Review of Data-to-Text NLGcited by 4
  26. 2022INFINITY: A Simple Yet Effective Unsupervised Framework for Graph-Text Mutual Conversioncited by 3
  27. 2022TextBox 2.0: A Text Generation Library with Pre-trained Language Modelscited by 3
  28. 2025Can LLMs Interpret and Leverage Structured Linguistic Representations? A Case Study with AMRscited by 3
  29. 2021UNITER-Based Situated Coreference Resolution with Rich Multimodal Inputcited by 2
  30. 2022Curriculum-Based Self-Training Makes Better Few-Shot Learners for Data-to-Text Generationcited by 2
  31. 2023Few-Shot Data-to-Text Generation via Unified Representation and Multi-Source Learningcited by 2
  32. 2023ASPIRO: Any-shot Structured Parsing-error-Induced ReprOmpting for Consistent Data-to-Text Generationcited by 2
  33. 2024GraSAME: Injecting Token-Level Structural Information to Pretrained Language Models via Graph-guided Self-Attention Mechanismcited by 2
  34. 2024MGSA: Multi-Granularity Graph Structure Attention for Knowledge Graph-to-Text Generationcited by 2
  35. 2025BALI: Enhancing Biomedical Language Representations through Knowledge Graph and Language Model Alignmentcited by 2
  36. 2022Syntax Controlled Knowledge Graph-to-Text Generation with Order and Semantic Consistencycited by 1
  37. 2023Learning from Multiple Sources for Data-to-Text and Text-to-Datacited by 1
  38. 2023Knowledge Graph for NLG in the context of conversational agentscited by 1

50 most cited of 58 All 58 (JSON)