# Transformers are Short Text Classifiers: A Study of Inductive Short Text Classifiers on Benchmarks and Real-world Datasets

arXiv: https://arxiv.org/abs/2211.16878 · HTML: https://mlsanity.com/paper/2211.16878/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 53 references that are in the map.
- First posted on arXiv 30 Nov 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Pitfalls of Graph Neural Network Evaluation](https://mlsanity.com/paper/1811.05868/) (2018)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/1809.05679/) (2018)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [A Convolutional Neural Network for Modelling Sentences](https://mlsanity.com/paper/1404.2188/) (2014)
- [Recurrent Neural Network for Text Classification with Multi-Task Learning](https://mlsanity.com/paper/1605.05101/) (2016)
- [A Survey on Text Classification: From Shallow to Deep Learning](https://mlsanity.com/paper/2008.00364/) (2020)
- [Every Document Owns Its Structure: Inductive Text Classification via Graph Neural Networks](https://mlsanity.com/paper/2004.13826/) (2020)
- [Be More with Less: Hypergraph Attention Networks for Inductive Text Classification](https://mlsanity.com/paper/2011.00387/) (2020)
- [BertGCN: Transductive Text Classification by Combining GCN and BERT](https://mlsanity.com/paper/2105.05727/) (2021)
- [Graph-MLP: Node Classification without Message Passing in Graph](https://mlsanity.com/paper/2106.04051/) (2021)
- [MAVE: A Product Dataset for Multi-source Attribute Value Extraction](https://mlsanity.com/paper/2112.08663/) (2021)
- [Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLP](https://mlsanity.com/paper/2109.03777/) (2021)
- [InducT-GCN: Inductive Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/2206.00265/) (2022)
- [Hierarchical Heterogeneous Graph Representation Learning for Short Text Classification](https://mlsanity.com/paper/2111.00180/) (2021)
- [Using Deep Learning for Title-Based Semantic Subject Indexing to Reach Competitive Performance to Full-Text](https://mlsanity.com/paper/1801.06717/) (2018)
- [MGNC-CNN: A Simple Approach to Exploiting Multiple Word Embeddings for Sentence Classification](https://mlsanity.com/paper/1603.00968/) (2016)
- [Using Titles vs. Full-text as Source for Automated Semantic Document Annotation](https://mlsanity.com/paper/1705.05311/) (2017)

## Built on by (most cited first, 12 of 12)

- [Efficient Detection of Toxic Prompts in Large Language Models](https://mlsanity.com/paper/2408.11727/) (2024)
- [Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification](https://mlsanity.com/paper/2406.01283/) (2024)
- [DistillSeq: A Framework for Safety Alignment Testing in Large Language Models using Knowledge Distillation](https://mlsanity.com/paper/2407.10106/) (2024)
- [Chinese Fine-Grained Financial Sentiment Analysis with Large Language Models](https://mlsanity.com/paper/2306.14096/) (2023)
- [Beyond Confusion: A Fine-grained Dialectical Examination of Human Activity Recognition Benchmark Datasets](https://mlsanity.com/paper/2412.09037/) (2024)
- [Aspect-Based Sentiment Analysis for Open-Ended HR Survey Responses](https://mlsanity.com/paper/2402.04812/) (2024)
- [Combining Denoising Autoencoders with Contrastive Learning to fine-tune Transformer Models](https://mlsanity.com/paper/2405.14437/) (2024)
- [Multi-label classification of open-ended questions with BERT](https://mlsanity.com/paper/2304.02945/) (2023)
- [Gpachov at CheckThat! 2023: A Diverse Multi-Approach Ensemble for Subjectivity Detection in News Articles](https://mlsanity.com/paper/2309.06844/) (2023)
- [AllTogether: Investigating the Efficacy of Spliced Prompt for Web Navigation using Large Language Models](https://mlsanity.com/paper/2310.18331/) (2023)
- [Integrated ensemble of BERT- and features-based models for authorship attribution in Japanese literary works](https://mlsanity.com/paper/2504.08527/) (2025)
- [Leveraging Knowledge Graph Embeddings to Enhance Contextual Representations for Relation Extraction](https://mlsanity.com/paper/2306.04203/) (2023)

## Abstract (arXiv)

Short text classification is a crucial and challenging aspect of Natural Language Processing. For this reason, there are numerous highly specialized short text classifiers. However, in recent short text research, State of the Art (SOTA) methods for traditional text classification, particularly the pure use of Transformers, have been unexploited. In this work, we examine the performance of a variety of short text classifiers as well as the top performing traditional text classifier. We further investigate the effects on two new real-world short text datasets in an effort to address the issue of becoming overly dependent on benchmark datasets with a limited number of characteristics. Our experiments unambiguously demonstrate that Transformers achieve SOTA accuracy on short text classification tasks, raising the question of whether specialized short text techniques are necessary.