# Comparing BERT against traditional machine learning text classification

arXiv: https://arxiv.org/abs/2005.13012 · HTML: https://mlsanity.com/paper/2005.13012/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2020.

## Built on by (most cited first, 30 of 46)

- [DFX: A Low-latency Multi-FPGA Appliance for Accelerating Transformer-based Text Generation](https://mlsanity.com/paper/2209.10797/) (2022)
- [Towards Modeling Learner Performance with Large Language Models](https://mlsanity.com/paper/2403.14661/) (2024)
- [Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLP](https://mlsanity.com/paper/2109.03777/) (2021)
- [From Text to Emotion: Unveiling the Emotion Annotation Capabilities of LLMs](https://mlsanity.com/paper/2408.17026/) (2024)
- [Comparative Study of Long Document Classification](https://mlsanity.com/paper/2111.00702/) (2021)
- [DEPTWEET: A Typology for Social Media Texts to Detect Depression Severities](https://mlsanity.com/paper/2210.05372/) (2022)
- [Transforming Fake News: Robust Generalisable News Classification Using Transformers](https://mlsanity.com/paper/2109.09796/) (2021)
- [Robustness and Confounders in the Demographic Alignment of LLMs with Human Perceptions of Offensiveness](https://mlsanity.com/paper/2411.08977/) (2024)
- [A Comparative Analysis of Transformer and LSTM Models for Detecting Suicidal Ideation on Reddit](https://mlsanity.com/paper/2411.15404/) (2024)
- [Vulnerability of LLMs to Vertically Aligned Text Manipulations](https://mlsanity.com/paper/2410.20016/) (2024)
- [LLM Content Moderation and User Satisfaction: Evidence from Response Refusals in Chatbot Arena](https://mlsanity.com/paper/2501.03266/) (2025)
- [Knowledge Distillation of LLM for Automatic Scoring of Science Education Assessments](https://mlsanity.com/paper/2312.15842/) (2023)
- [CultureBERT: Measuring Corporate Culture With Transformer-Based Language Models](https://mlsanity.com/paper/2212.00509/) (2022)
- [TweetDrought: A Deep-Learning Drought Impacts Recognizer based on Twitter Data](https://mlsanity.com/paper/2212.04001/) (2022)
- [Fine-tuning ClimateBert transformer with ClimaText for the disclosure analysis of climate-related financial risks](https://mlsanity.com/paper/2303.13373/) (2023)
- [A Novel Patent Similarity Measurement Methodology: Semantic Distance and Technological Distance](https://mlsanity.com/paper/2303.16767/) (2023)
- [Harnessing the Power of Hugging Face Transformers for Predicting Mental Health Disorders in Social Networks](https://mlsanity.com/paper/2306.16891/) (2023)
- [SpamDam: Towards Privacy-Preserving and Adversary-Resistant SMS Spam Detection](https://mlsanity.com/paper/2404.09481/) (2024)
- [Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review](https://mlsanity.com/paper/2510.17892/) (2025)
- [Textual Data Distributions: Kullback Leibler Textual Distributions Contrasts on GPT-2 Generated Texts, with Supervised, Unsupervised Learning on Vaccine & Market Topics & Sentiment](https://mlsanity.com/paper/2107.02025/) (2021)
- [Evaluating BERT-based Pre-training Language Models for Detecting Misinformation](https://mlsanity.com/paper/2203.07731/) (2022)
- [MAPLE: Mobile App Prediction Leveraging Large Language Model Embeddings](https://mlsanity.com/paper/2309.08648/) (2023)
- [Robustifying Language Models with Test-Time Adaptation](https://mlsanity.com/paper/2310.19177/) (2023)
- [KDH-MLTC: Knowledge Distillation for Healthcare Multi-Label Text Classification](https://mlsanity.com/paper/2505.07162/) (2025)
- [Sentiment Analysis of the COVID-related r/Depression Posts](https://mlsanity.com/paper/2108.06215/) (2021)
- [StonkBERT: Can Language Models Predict Medium-Run Stock Price Movements?](https://mlsanity.com/paper/2202.02268/) (2022)
- [Are We Really Making Much Progress in Text Classification? A Comparative Review](https://mlsanity.com/paper/2204.03954/) (2022)
- [Classification of Natural Language Processing Techniques for Requirements Engineering](https://mlsanity.com/paper/2204.04282/) (2022)
- [A Twitter-Driven Deep Learning Mechanism for the Determination of Vehicle Hijacking Spots in Cities](https://mlsanity.com/paper/2208.10280/) (2022)
- [KhabarChin: Automatic Detection of Important News in the Persian Language](https://mlsanity.com/paper/2312.03361/) (2023)

## Abstract (arXiv)

The BERT model has arisen as a popular state-of-the-art machine learning model in the recent years that is able to cope with multiple NLP tasks such as supervised text classification without human supervision. Its flexibility to cope with any type of corpus delivering great results has make this approach very popular not only in academia but also in the industry. Although, there are lots of different approaches that have been used throughout the years with success. In this work, we first present BERT and include a little review on classical NLP approaches. Then, we empirically test with a suite of experiments dealing different scenarios the behaviour of BERT against the traditional TF-IDF vocabulary fed to machine learning algorithms. Our purpose of this work is to add empirical evidence to support or refuse the use of BERT as a default on NLP tasks. Experiments show the superiority of BERT and its independence of features of the NLP problem such as the language of the text adding empirical evidence to use BERT as a default technique to be used in NLP problems.