# Transformer-based Language Model Fine-tuning Methods for COVID-19 Fake News Detection

arXiv: https://arxiv.org/abs/2101.05509 · HTML: https://mlsanity.com/paper/2101.05509/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 14 Jan 2021.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Adversarial Training Methods for Semi-Supervised Text Classification](https://mlsanity.com/paper/1605.07725/) (2016)
- [FreeLB: Enhanced Adversarial Training for Natural Language Understanding](https://mlsanity.com/paper/1909.11764/) (2019)
- [Generating Adversarial Examples with Adversarial Networks](https://mlsanity.com/paper/1801.02610/) (2018)
- [COVID-Twitter-BERT: A Natural Language Processing Model to Analyse COVID-19 Content on Twitter](https://mlsanity.com/paper/2005.07503/) (2020)
- [Fighting an Infodemic: COVID-19 Fake News Dataset](https://mlsanity.com/paper/2011.03327/) (2020)
- [FakeCovid -- A Multilingual Cross-domain Fact Check News Dataset for COVID-19](https://mlsanity.com/paper/2006.11343/) (2020)
- [Heated-Up Softmax Embedding](https://mlsanity.com/paper/1809.04157/) (2018)

## Built on by (most cited first, 15 of 15)

- [ChatGPT Perpetuates Gender Bias in Machine Translation and Ignores Non-Gendered Pronouns: Findings across Bengali and Five other Low-Resource Languages](https://mlsanity.com/paper/2305.10510/) (2023)
- [Improving Paraphrase Detection with the Adversarial Paraphrasing Task](https://mlsanity.com/paper/2106.07691/) (2021)
- [Knowledge Fusion By Evolving Weights of Language Models](https://mlsanity.com/paper/2406.12208/) (2024)
- [What goes on inside rumour and non-rumour tweets and their reactions: A Psycholinguistic Analyses](https://mlsanity.com/paper/2112.03003/) (2021)
- [NLP-CUET@LT-EDI-EACL2021: Multilingual Code-Mixed Hope Speech Detection using Cross-lingual Representation Learner](https://mlsanity.com/paper/2103.00464/) (2021)
- [Emotion Classification in a Resource Constrained Language Using Transformer-based Approach](https://mlsanity.com/paper/2104.08613/) (2021)
- [MiDAS: Multi-integrated Domain Adaptive Supervision for Fake News Detection](https://mlsanity.com/paper/2205.09817/) (2022)
- [Findings of Factify 2: Multimodal Fake News Detection](https://mlsanity.com/paper/2307.10475/) (2023)
- [Adversarial Style Augmentation via Large Language Model for Robust Fake News Detection](https://mlsanity.com/paper/2406.11260/) (2024)
- [Similarity-Aware Multimodal Prompt Learning for Fake News Detection](https://mlsanity.com/paper/2304.04187/) (2023)
- [Evaluating Generalizability of Fine-Tuned Models for Fake News Detection](https://mlsanity.com/paper/2205.07154/) (2022)
- [Are You Misinformed? A Study of Covid-Related Fake News in Bengali on Facebook](https://mlsanity.com/paper/2203.11669/) (2022)
- [Automated Evidence Collection for Fake News Detection](https://mlsanity.com/paper/2112.06507/) (2021)
- [Speech Detection Task Against Asian Hate: BERT the Central, While Data-Centric Studies the Crucial](https://mlsanity.com/paper/2206.02114/) (2022)
- [A Comparative Analysis of the COVID-19 Infodemic in English and Chinese: Insights from Social Media Textual Data](https://mlsanity.com/paper/2311.08001/) (2023)

## Abstract (arXiv)

With the pandemic of COVID-19, relevant fake news is spreading all over the sky throughout the social media. Believing in them without discrimination can cause great trouble to people's life. However, universal language models may perform weakly in these fake news detection for lack of large-scale annotated data and sufficient semantic understanding of domain-specific knowledge. While the model trained on corresponding corpora is also mediocre for insufficient learning. In this paper, we propose a novel transformer-based language model fine-tuning approach for these fake news detection. First, the token vocabulary of individual model is expanded for the actual semantics of professional phrases. Second, we adapt the heated-up softmax loss to distinguish the hard-mining samples, which are common for fake news because of the disambiguation of short text. Then, we involve adversarial training to improve the model's robustness. Last, the predicted features extracted by universal language model RoBERTa and domain-specific model CT-BERT are fused by one multiple layer perception to integrate fine-grained and high-level specific representations. Quantitative experimental results evaluated on existing COVID-19 fake news dataset show its superior performances compared to the state-of-the-art methods among various evaluation metrics. Furthermore, the best weighted average F1 score achieves 99.02%.