# Exploring Text-transformers in AAAI 2021 Shared Task: COVID-19 Fake News Detection in English

arXiv: https://arxiv.org/abs/2101.02359 · HTML: https://mlsanity.com/paper/2101.02359/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 20 references that are in the map.
- First posted on arXiv 7 Jan 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [ERNIE: Enhanced Representation through Knowledge Integration](https://mlsanity.com/paper/1904.09223/) (2019)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [Fighting an Infodemic: COVID-19 Fake News Dataset](https://mlsanity.com/paper/2011.03327/) (2020)
- [Early Detection of Fake News by Utilizing the Credibility of News, Publishers, and Users Based on Weakly Supervised Learning](https://mlsanity.com/paper/2012.04233/) (2020)
- [Fake News Detection in Social Media using Graph Neural Networks and NLP Techniques: A COVID-19 Use-case](https://mlsanity.com/paper/2012.07517/) (2020)
- [Fake news agenda in the era of COVID-19: Identifying trends through fact-checking content](https://mlsanity.com/paper/2012.11004/) (2020)

## Built on by (most cited first, 11 of 11)

- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [Categorising Fine-to-Coarse Grained Misinformation: An Empirical Study of COVID-19 Infodemic](https://mlsanity.com/paper/2106.11702/) (2021)
- [Combating Health Misinformation in Social Media: Characterization, Detection, Intervention, and Open Issues](https://mlsanity.com/paper/2211.05289/) (2022)
- [Automated multilingual detection of Pro-Kremlin propaganda in newspapers and Telegram posts](https://mlsanity.com/paper/2301.10604/) (2023)
- [A Heuristic-driven Uncertainty based Ensemble Framework for Fake News Detection in Tweets and News Articles](https://mlsanity.com/paper/2104.01791/) (2021)
- [Low Resource Style Transfer via Domain Adaptive Meta Learning](https://mlsanity.com/paper/2205.12475/) (2022)
- [Beyond Detection: Exploring Evidence-based Multi-Agent Debate for Misinformation Intervention and Persuasion](https://mlsanity.com/paper/2511.07267/) (2025)
- [Multilingual Models for Check-Worthy Social Media Posts Detection](https://mlsanity.com/paper/2408.06737/) (2024)
- [A Multi-Branch Feature Fusion Approach for Health Misinformation Detection and Propagation](https://mlsanity.com/paper/2609.00403/) (2026)
- [Automated Evidence Collection for Fake News Detection](https://mlsanity.com/paper/2112.06507/) (2021)
- [Health Misinformation in Social Networks: A Survey of IT Approaches](https://mlsanity.com/paper/2410.18670/) (2024)

## Abstract (arXiv)

In this paper, we describe our system for the AAAI 2021 shared task of COVID-19 Fake News Detection in English, where we achieved the 3rd position with the weighted F1 score of 0.9859 on the test set. Specifically, we proposed an ensemble method of different pre-trained language models such as BERT, Roberta, Ernie, etc. with various training strategies including warm-up,learning rate schedule and k-fold cross-validation. We also conduct an extensive analysis of the samples that are not correctly classified. The code is available at:https://github.com/archersama/3rd-solution-COVID19-Fake-News-Detection-in-English.