# Challenges in Pre-Training Graph Neural Networks for Context-Based Fake News Detection: An Evaluation of Current Strategies and Resource Limitations

arXiv: https://arxiv.org/abs/2402.18179 · HTML: https://mlsanity.com/paper/2402.18179/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 47 references that are in the map.
- First posted on arXiv 28 Feb 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [GPT-GNN: Generative Pre-Training of Graph Neural Networks](https://mlsanity.com/paper/2006.15437/) (2020)
- [FakeNewsNet: A Data Repository with News Content, Social Context and Spatialtemporal Information for Studying Fake News on Social Media](https://mlsanity.com/paper/1809.01286/) (2018)
- [GCAN: Graph-aware Co-Attention Networks for Explainable Fake News Detection on Social Media](https://mlsanity.com/paper/2004.11648/) (2020)
- [FANG: Leveraging Social Context for Fake News Detection Using Graph Representation](https://mlsanity.com/paper/2008.07939/) (2020)
- [Beyond News Contents: The Role of Social Context for Fake News Detection](https://mlsanity.com/paper/1712.07709/) (2017)
- [Predicting Factuality of Reporting and Bias of News Media Sources](https://mlsanity.com/paper/1810.01765/) (2018)
- [All-in-one: Multi-task Learning for Rumour Verification](https://mlsanity.com/paper/1806.03713/) (2018)
- [Pre-Training Graph Neural Networks for Cold-Start Users and Items Representation](https://mlsanity.com/paper/2012.07064/) (2020)
- [Self-supervised Heterogeneous Graph Pre-training Based on Structural Clustering](https://mlsanity.com/paper/2210.10462/) (2022)
- [Graph-based Modeling of Online Communities for Fake News Detection](https://mlsanity.com/paper/2008.06274/) (2020)
- [Applying Automatic Text Summarization for Fake News Detection](https://mlsanity.com/paper/2204.01841/) (2022)
- [Fake News Detection with Heterogeneous Transformer](https://mlsanity.com/paper/2205.03100/) (2022)
- [Interactively Learning Social Media Representations Improves News Source Factuality Detection](https://mlsanity.com/paper/2309.14966/) (2023)
- [Exploring Fake News Detection with Heterogeneous Social Media Context Graphs](https://mlsanity.com/paper/2212.06560/) (2022)

## Built on by (most cited first, 2 of 2)

- [MGA-VQA: Secure and Interpretable Graph-Augmented Visual Question Answering with Memory-Guided Protection Against Unauthorized Knowledge Use](https://mlsanity.com/paper/2511.17881/) (2025)
- [Community Norms in the Spotlight: Enabling Task-Agnostic Unsupervised Pre-Training to Benefit Online Social Media](https://mlsanity.com/paper/2602.02525/) (2026)

## Abstract (arXiv)

Pre-training of neural networks has recently revolutionized the field of Natural Language Processing (NLP) and has before demonstrated its effectiveness in computer vision. At the same time, advances around the detection of fake news were mainly driven by the context-based paradigm, where different types of signals (e.g. from social media) form graph-like structures that hold contextual information apart from the news article to classify. We propose to merge these two developments by applying pre-training of Graph Neural Networks (GNNs) in the domain of context-based fake news detection. Our experiments provide an evaluation of different pre-training strategies for graph-based misinformation detection and demonstrate that transfer learning does currently not lead to significant improvements over training a model from scratch in the domain. We argue that a major current issue is the lack of suitable large-scale resources that can be used for pre-training.