# VGCN-BERT: Augmenting BERT with Graph Embedding for Text Classification

arXiv: https://arxiv.org/abs/2004.05707 · HTML: https://mlsanity.com/paper/2004.05707/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2020.

## Built on by (most cited first, 30 of 35)

- [BertGCN: Transductive Text Classification by Combining GCN and BERT](https://mlsanity.com/paper/2105.05727/) (2021)
- [AMR Parsing with Action-Pointer Transformer](https://mlsanity.com/paper/2104.14674/) (2021)
- [PokeMQA: Programmable knowledge editing for Multi-hop Question Answering](https://mlsanity.com/paper/2312.15194/) (2023)
- [Graph Pooling via Coarsened Graph Infomax](https://mlsanity.com/paper/2105.01275/) (2021)
- [Graph Neural Networks for Text Classification: A Survey](https://mlsanity.com/paper/2304.11534/) (2023)
- [WGCN: Graph Convolutional Networks with Weighted Structural Features](https://mlsanity.com/paper/2104.14060/) (2021)
- [ICDBigBird: A Contextual Embedding Model for ICD Code Classification](https://mlsanity.com/paper/2204.10408/) (2022)
- [Constructing Flow Graphs from Procedural Cybersecurity Texts](https://mlsanity.com/paper/2105.14357/) (2021)
- [Ethereum Fraud Detection via Joint Transaction Language Model and Graph Representation Learning](https://mlsanity.com/paper/2409.07494/) (2024)
- [Cross-utterance Reranking Models with BERT and Graph Convolutional Networks for Conversational Speech Recognition](https://mlsanity.com/paper/2106.06922/) (2021)
- [Efficacy of BERT embeddings on predicting disaster from Twitter data](https://mlsanity.com/paper/2108.10698/) (2021)
- [Differentiable Retrieval Augmentation via Generative Language Modeling for E-commerce Query Intent Classification](https://mlsanity.com/paper/2308.09308/) (2023)
- [Integrating Structural and Semantic Signals in Text-Attributed Graphs with BiGTex](https://mlsanity.com/paper/2504.12474/) (2025)
- [LMAE4Eth: Generalizable and Robust Ethereum Fraud Detection by Exploring Transaction Semantics and Masked Graph Embedding](https://mlsanity.com/paper/2509.03939/) (2025)
- [Navigating the Kaleidoscope of COVID-19 Misinformation Using Deep Learning](https://mlsanity.com/paper/2110.15703/) (2021)
- [SocialBERT -- Transformers for Online SocialNetwork Language Modelling](https://mlsanity.com/paper/2111.07148/) (2021)
- [Explainable Identification of Hate Speech towards Islam using Graph Neural Networks](https://mlsanity.com/paper/2311.04916/) (2023)
- [Jensen-Shannon Divergence Message-Passing for Rich-Text Graph Representation Learning](https://mlsanity.com/paper/2512.20094/) (2025)
- [An Analysis of a BERT Deep Learning Strategy on a Technology Assisted Review Task](https://mlsanity.com/paper/2104.08340/) (2021)
- [PG2Net: Personalized and Group Preferences Guided Network for Next Place Prediction](https://mlsanity.com/paper/2110.08266/) (2021)
- [Leveraging Dependency Grammar for Fine-Grained Offensive Language Detection using Graph Convolutional Networks](https://mlsanity.com/paper/2205.13164/) (2022)
- [UPB at SemEval-2022 Task 5: Enhancing UNITER with Image Sentiment and Graph Convolutional Networks for Multimedia Automatic Misogyny Identification](https://mlsanity.com/paper/2205.14769/) (2022)
- [Sentence Ambiguity, Grammaticality and Complexity Probes](https://mlsanity.com/paper/2210.06928/) (2022)
- [ViCGCN: Graph Convolutional Network with Contextualized Language Models for Social Media Mining in Vietnamese](https://mlsanity.com/paper/2309.02902/) (2023)
- [Investigating the Impact of Semi-Supervised Methods with Data Augmentation on Offensive Language Detection in Romanian Language](https://mlsanity.com/paper/2407.20076/) (2024)
- [Text Classification using Graph Convolutional Networks: A Comprehensive Survey](https://mlsanity.com/paper/2410.09399/) (2024)
- [Improved Customer Transaction Classification using Semi-Supervised Knowledge Distillation](https://mlsanity.com/paper/2102.07635/) (2021)
- [RubCSG at SemEval-2022 Task 5: Ensemble learning for identifying misogynous MEMEs](https://mlsanity.com/paper/2204.03953/) (2022)
- [Knowledge Graph Fusion for Language Model Fine-tuning](https://mlsanity.com/paper/2206.14574/) (2022)
- [Word Grounded Graph Convolutional Network](https://mlsanity.com/paper/2305.06434/) (2023)

## Abstract (arXiv)

Much progress has been made recently on text classification with methods based on neural networks. In particular, models using attention mechanism such as BERT have shown to have the capability of capturing the contextual information within a sentence or document. However, their ability of capturing the global information about the vocabulary of a language is more limited. This latter is the strength of Graph Convolutional Networks (GCN). In this paper, we propose VGCN-BERT model which combines the capability of BERT with a Vocabulary Graph Convolutional Network (VGCN). Local information and global information interact through different layers of BERT, allowing them to influence mutually and to build together a final representation for classification. In our experiments on several text classification datasets, our approach outperforms BERT and GCN alone, and achieve higher effectiveness than that reported in previous studies.