# Linear Classifier: An Often-Forgotten Baseline for Text Classification

arXiv: https://arxiv.org/abs/2306.07111 · HTML: https://mlsanity.com/paper/2306.07111/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 28 references that are in the map.
- First posted on arXiv 12 Jun 2023.

## Builds on (most cited first, 5 of 5)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [PECOS: Prediction for Enormous and Correlated Output Spaces](https://mlsanity.com/paper/2010.05878/) (2020)
- [A Comparison of SVM against Pre-trained Language Models (PLMs) for Text Classification Tasks](https://mlsanity.com/paper/2211.02563/) (2022)
- [On the Use of Unrealistic Predictions in Hundreds of Papers Evaluating Graph Representations](https://mlsanity.com/paper/2112.04274/) (2021)

## Built on by (most cited first, 7 of 7)

- [Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning](https://mlsanity.com/paper/2602.04998/) (2026)
- [A Survey of Classification Tasks and Approaches for Legal Contracts](https://mlsanity.com/paper/2507.21108/) (2025)
- [Do You Really Need a GPU to Guard Your LLM? CPU-Class Classifiers and Multi-Stage Pipelines for Safety Enforcement at Scale](https://mlsanity.com/paper/2512.19011/) (2025)
- [Deep dive into language traits of AI-generated Abstracts](https://mlsanity.com/paper/2312.10617/) (2023)
- [Enriching Social Science Research via Survey Item Linking](https://mlsanity.com/paper/2412.15831/) (2024)
- [Multilingual Lexical Feature Analysis of Spoken Language for Predicting Major Depression Symptom Severity](https://mlsanity.com/paper/2511.07011/) (2025)
- [Automatic Cardiac Risk Management Classification using large-context Electronic Patients Health Records](https://mlsanity.com/paper/2603.09685/) (2026)

## Abstract (arXiv)

Large-scale pre-trained language models such as BERT are popular solutions for text classification. Due to the superior performance of these advanced methods, nowadays, people often directly train them for a few epochs and deploy the obtained model. In this opinion paper, we point out that this way may only sometimes get satisfactory results. We argue the importance of running a simple baseline like linear classifiers on bag-of-words features along with advanced methods. First, for many text data, linear methods show competitive performance, high efficiency, and robustness. Second, advanced models such as BERT may only achieve the best results if properly applied. Simple baselines help to confirm whether the results of advanced models are acceptable. Our experimental results fully support these points.