mlsanityData to 14 Oct 2026

Transformers are Short Text Classifiers: A Study of Inductive Short Text Classifiers on Benchmarks and Real-world Datasets

arXiv 2211.16878 ↗First posted 30 Nov 2022cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Short text classification is a crucial and challenging aspect of Natural Language Processing. For this reason, there are numerous highly specialized short text classifiers. However, in recent short text research, State of the Art (SOTA) methods for traditional text classification, particularly the pure use of Transformers, have been unexploited. In this work, we examine the performance of a variety of short text classifiers as well as the top performing traditional text classifier. We further investigate the effects on two new real-world short text datasets in an effort to address the issue of becoming overly dependent on benchmark datasets with a limited number of characteristics. Our experiments unambiguously demonstrate that Transformers achieve SOTA accuracy on short text classification tasks, raising the question of whether specialized short text techniques are necessary.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 26 of its 53 references that are in the map. First posted on arXiv 30 Nov 2022.

Builds on

26 of 53 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  7. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,127
  8. 2018Pitfalls of Graph Neural Network Evaluationcited by 1,007
  9. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  10. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  11. 2018Graph Convolutional Networks for Text Classificationcited by 345
  12. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 231
14 more
  1. 2014A Convolutional Neural Network for Modelling Sentencescited by 163
  2. 2016Recurrent Neural Network for Text Classification with Multi-Task Learningcited by 144
  3. 2020A Survey on Text Classification: From Shallow to Deep Learningcited by 80
  4. 2020Every Document Owns Its Structure: Inductive Text Classification via Graph Neural Networkscited by 77
  5. 2020Be More with Less: Hypergraph Attention Networks for Inductive Text Classificationcited by 74
  6. 2021BertGCN: Transductive Text Classification by Combining GCN and BERTcited by 74
  7. 2021Graph-MLP: Node Classification without Message Passing in Graphcited by 73
  8. 2021MAVE: A Product Dataset for Multi-source Attribute Value Extractioncited by 53
  9. 2021Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLPcited by 17
  10. 2022InducT-GCN: Inductive Graph Convolutional Networks for Text Classificationcited by 15
  11. 2021Hierarchical Heterogeneous Graph Representation Learning for Short Text Classificationcited by 13
  12. 2018Using Deep Learning for Title-Based Semantic Subject Indexing to Reach Competitive Performance to Full-Textcited by 10
  13. 2016MGNC-CNN: A Simple Approach to Exploiting Multiple Word Embeddings for Sentence Classificationcited by 4
  14. 2017Using Titles vs. Full-text as Source for Automated Semantic Document Annotationcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 6; 2025: 1.

12 papers.

  1. 2024Efficient Detection of Toxic Prompts in Large Language Modelscited by 13
  2. 2024Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classificationcited by 7
  3. 2024DistillSeq: A Framework for Safety Alignment Testing in Large Language Models using Knowledge Distillationcited by 5
  4. 2023Chinese Fine-Grained Financial Sentiment Analysis with Large Language Modelscited by 4
  5. 2024Beyond Confusion: A Fine-grained Dialectical Examination of Human Activity Recognition Benchmark Datasetscited by 4
  6. 2024Aspect-Based Sentiment Analysis for Open-Ended HR Survey Responsescited by 3
  7. 2024Combining Denoising Autoencoders with Contrastive Learning to fine-tune Transformer Modelscited by 2
  8. 2023Multi-label classification of open-ended questions with BERTcited by 1
  9. 2023Gpachov at CheckThat! 2023: A Diverse Multi-Approach Ensemble for Subjectivity Detection in News Articlescited by 1
  10. 2023AllTogether: Investigating the Efficacy of Spliced Prompt for Web Navigation using Large Language Modelscited by 1
  11. 2025Integrated ensemble of BERT- and features-based models for authorship attribution in Japanese literary workscited by 1
  12. 2023Leveraging Knowledge Graph Embeddings to Enhance Contextual Representations for Relation Extractioncited by 0