# Evaluating Generalizability of Fine-Tuned Models for Fake News Detection

arXiv: https://arxiv.org/abs/2205.07154 · HTML: https://mlsanity.com/paper/2205.07154/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 15 May 2022.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [No Fuss Distance Metric Learning using Proxies](https://mlsanity.com/paper/1703.07464/) (2017)
- [What Happens To BERT Embeddings During Fine-tuning?](https://mlsanity.com/paper/2004.14448/) (2020)
- [An introduction to domain adaptation and transfer learning](https://mlsanity.com/paper/1812.11806/) (2018)
- [Machine Learning with a Reject Option: A survey](https://mlsanity.com/paper/2107.11277/) (2021)
- [ProxyNCA++: Revisiting and Revitalizing Proxy Neighborhood Component Analysis](https://mlsanity.com/paper/2004.01113/) (2020)
- [COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter](https://mlsanity.com/paper/2005.07503/) (2020)
- [What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning](https://mlsanity.com/paper/1911.03090/) (2019)
- [CoAID: COVID-19 Healthcare Misinformation Dataset](https://mlsanity.com/paper/2006.00885/) (2020)
- [End-to-End Weak Supervision](https://mlsanity.com/paper/2107.02233/) (2021)
- [Characterizing COVID-19 Misinformation Communities Using a Novel Twitter Dataset](https://mlsanity.com/paper/2008.00791/) (2020)
- [Shoring Up the Foundations: Fusing Model Embeddings and Weak Supervision](https://mlsanity.com/paper/2203.13270/) (2022)
- [Accurate estimation of influenza epidemics using Google search data via ARGO](https://mlsanity.com/paper/1505.00864/) (2015)
- [Transformer-based Language Model Fine-tuning Methods for COVID-19 Fake News Detection](https://mlsanity.com/paper/2101.05509/) (2021)
- [A Stance Data Set on Polarized Conversations on Twitter about the Efficacy of Hydroxychloroquine as a Treatment for COVID-19](https://mlsanity.com/paper/2009.01188/) (2020)
- [Towards Lingua Franca Named Entity Recognition with BERT](https://mlsanity.com/paper/1912.01389/) (2019)
- [Model Generalization on COVID-19 Fake News Detection](https://mlsanity.com/paper/2101.03841/) (2021)
- [Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection](https://mlsanity.com/paper/2111.07819/) (2021)
- [LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation](https://mlsanity.com/paper/2103.06418/) (2021)
- [A Heuristic-driven Uncertainty based Ensemble Framework for Fake News Detection in Tweets and News Articles](https://mlsanity.com/paper/2104.01791/) (2021)
- [Explainable Misinformation Detection Across Multiple Social Media Platforms](https://mlsanity.com/paper/2203.11724/) (2022)
- [Convergence of online k-means](https://mlsanity.com/paper/2202.10640/) (2022)
- [Concept Drift Adaptive Physical Event Detection for Social Media Streams](https://mlsanity.com/paper/1911.05494/) (2019)
- [Exploring BERT Parameter Efficiency on the Stanford Question Answering Dataset v2.0](https://mlsanity.com/paper/2002.10670/) (2020)

## Built on by (most cited first, 3 of 3)

- [MiDAS: Multi-integrated Domain Adaptive Supervision for Fake News Detection](https://mlsanity.com/paper/2205.09817/) (2022)
- [Time-Aware Datasets are Adaptive Knowledgebases for the New Normal](https://mlsanity.com/paper/2211.12508/) (2022)
- [Continuously Reliable Detection of New-Normal Misinformation: Semantic Masking and Contrastive Smoothing in High-Density Latent Regions](https://mlsanity.com/paper/2301.07981/) (2023)

## Abstract (arXiv)

The Covid-19 pandemic has caused a dramatic and parallel rise in dangerous misinformation, denoted an `infodemic' by the CDC and WHO. Misinformation tied to the Covid-19 infodemic changes continuously; this can lead to performance degradation of fine-tuned models due to concept drift. Degredation can be mitigated if models generalize well-enough to capture some cyclical aspects of drifted data. In this paper, we explore generalizability of pre-trained and fine-tuned fake news detectors across 9 fake news datasets. We show that existing models often overfit on their training dataset and have poor performance on unseen data. However, on some subsets of unseen data that overlap with training data, models have higher accuracy. Based on this observation, we also present KMeans-Proxy, a fast and effective method based on K-Means clustering for quickly identifying these overlapping subsets of unseen data. KMeans-Proxy improves generalizability on unseen fake news datasets by 0.1-0.2 f1-points across datasets. We present both our generalizability experiments as well as KMeans-Proxy to further research in tackling the fake news problem.