# BertGCN: Transductive Text Classification by Combining GCN and BERT

arXiv: https://arxiv.org/abs/2105.05727 · HTML: https://mlsanity.com/paper/2105.05727/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 50 references that are in the map.
- First posted on arXiv 12 May 2021.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [MolGAN: An implicit generative model for small molecular graphs](https://mlsanity.com/paper/1805.11973/) (2018)
- [Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/1809.05679/) (2018)
- [Learning Deep Generative Models of Graphs](https://mlsanity.com/paper/1803.03324/) (2018)
- [GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs](https://mlsanity.com/paper/1803.07294/) (2018)
- [Graph Convolution over Pruned Dependency Trees Improves Relation Extraction](https://mlsanity.com/paper/1809.10185/) (2018)
- [Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling](https://mlsanity.com/paper/1703.04826/) (2017)
- [PTE: Predictive Text Embedding through Large-scale Heterogeneous Text Networks](https://mlsanity.com/paper/1508.00200/) (2015)
- [Geometric Matrix Completion with Recurrent Multi-Graph Neural Networks](https://mlsanity.com/paper/1704.06803/) (2017)
- [Graph Convolutional Encoders for Syntax-aware Neural Machine Translation](https://mlsanity.com/paper/1704.04675/) (2017)
- [Joint Embedding of Words and Labels for Text Classification](https://mlsanity.com/paper/1805.04174/) (2018)
- [Text Level Graph Neural Network for Text Classification](https://mlsanity.com/paper/1910.02356/) (2019)
- [Question Answering by Reasoning Across Documents with Graph Convolutional Networks](https://mlsanity.com/paper/1808.09920/) (2018)
- [A C-LSTM Neural Network for Text Classification](https://mlsanity.com/paper/1511.08630/) (2015)
- [Tensor Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/2001.05313/) (2020)
- [VGCN-BERT: Augmenting BERT with Graph Embedding for Text Classification](https://mlsanity.com/paper/2004.05707/) (2020)
- [Deep Graph Convolutional Encoders for Structured Data to Text Generation](https://mlsanity.com/paper/1810.09995/) (2018)

## Built on by (most cited first, 30 of 74)

- [Text Classification via Large Language Models](https://mlsanity.com/paper/2305.08377/) (2023)
- [How would Stance Detection Techniques Evolve after the Launch of ChatGPT?](https://mlsanity.com/paper/2212.14548/) (2022)
- [OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages](https://mlsanity.com/paper/2110.05877/) (2021)
- [Pushing The Limit of LLM Capacity for Text Classification](https://mlsanity.com/paper/2402.07470/) (2024)
- [GNN-LM: Language Modeling based on Global Contexts via GNN](https://mlsanity.com/paper/2110.08743/) (2021)
- [ChatGraph: Interpretable Text Classification by Converting ChatGPT Knowledge to Graphs](https://mlsanity.com/paper/2305.03513/) (2023)
- [Knowledge-Aware Query Expansion with Large Language Models for Textual and Relational Retrieval](https://mlsanity.com/paper/2410.13765/) (2024)
- [Investigating Chain-of-thought with ChatGPT for Stance Detection on Social Media](https://mlsanity.com/paper/2304.03087/) (2023)
- [Infusing Knowledge from Wikipedia to Enhance Stance Detection](https://mlsanity.com/paper/2204.03839/) (2022)
- [Layer-wise Model Pruning based on Mutual Information](https://mlsanity.com/paper/2108.12594/) (2021)
- [Interpreting Deep Learning Models in Natural Language Processing: A Review](https://mlsanity.com/paper/2110.10470/) (2021)
- [Neural Additive Models for Location Scale and Shape: A Framework for Interpretable Neural Regression Beyond the Mean](https://mlsanity.com/paper/2301.11862/) (2023)
- [A Corpus for Understanding and Generating Moral Stories](https://mlsanity.com/paper/2204.09438/) (2022)
- [InducT-GCN: Inductive Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/2206.00265/) (2022)
- [Graph Neural Networks for Text Classification: A Survey](https://mlsanity.com/paper/2304.11534/) (2023)
- [Transformers are Short Text Classifiers: A Study of Inductive Short Text Classifiers on Benchmarks and Real-world Datasets](https://mlsanity.com/paper/2211.16878/) (2022)
- [Sentiment Analysis through LLM Negotiations](https://mlsanity.com/paper/2311.01876/) (2023)
- [kFolden: k-Fold Ensemble for Out-Of-Distribution Detection](https://mlsanity.com/paper/2108.12731/) (2021)
- [Finding the Law: Enhancing Statutory Article Retrieval via Graph Neural Networks](https://mlsanity.com/paper/2301.12847/) (2023)
- [Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Models](https://mlsanity.com/paper/2406.07001/) (2024)
- [Faster Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2112.08152/) (2021)
- [Ethereum Fraud Detection via Joint Transaction Language Model and Graph Representation Learning](https://mlsanity.com/paper/2409.07494/) (2024)
- [Graph Contrastive Learning via Cluster-refined Negative Sampling for Semi-supervised Text Classification](https://mlsanity.com/paper/2410.18130/) (2024)
- [Team Yao at Factify 2022: Utilizing Pre-trained Models and Co-attention Networks for Multi-Modal Fact Verification](https://mlsanity.com/paper/2201.11664/) (2022)
- [kNN-NER: Named Entity Recognition with Nearest Neighbor Search](https://mlsanity.com/paper/2203.17103/) (2022)
- [Localize, Retrieve and Fuse: A Generalized Framework for Free-Form Question Answering over Tables](https://mlsanity.com/paper/2309.11049/) (2023)
- [Contrastive Multi-graph Learning with Neighbor Hierarchical Sifting for Semi-supervised Text Classification](https://mlsanity.com/paper/2411.16787/) (2024)
- [Graph Learning and Its Advancements on Large Language Models: A Holistic Survey](https://mlsanity.com/paper/2212.08966/) (2022)
- [Continual Graph Convolutional Network for Text Classification](https://mlsanity.com/paper/2304.04152/) (2023)
- [Ladder-of-Thought: Using Knowledge as Steps to Elevate Stance Detection](https://mlsanity.com/paper/2308.16763/) (2023)

## Abstract (arXiv)

In this work, we propose BertGCN, a model that combines large scale pretraining and transductive learning for text classification. BertGCN constructs a heterogeneous graph over the dataset and represents documents as nodes using BERT representations. By jointly training the BERT and GCN modules within BertGCN, the proposed model is able to leverage the advantages of both worlds: large-scale pretraining which takes the advantage of the massive amount of raw data and transductive learning which jointly learns representations for both training data and unlabeled test data by propagating label influence through graph convolution. Experiments show that BertGCN achieves SOTA performances on a wide range of text classification datasets. Code is available at https://github.com/ZeroRin/BertGCN.