# Practical Text Classification With Large Pre-Trained Language Models

arXiv: https://arxiv.org/abs/1812.01207 · HTML: https://mlsanity.com/paper/1812.01207/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2018.

## Built on by (most cited first, 24 of 24)

- [Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook](https://mlsanity.com/paper/2310.10196/) (2023)
- [HowkGPT: Investigating the Detection of ChatGPT-generated University Student Homework through Context-Aware Perplexity Analysis](https://mlsanity.com/paper/2305.18226/) (2023)
- [A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice](https://mlsanity.com/paper/2404.16958/) (2024)
- [Knowledge Graph Completion Models are Few-shot Learners: An Empirical Study of Relation Labeling in E-commerce with LLMs](https://mlsanity.com/paper/2305.09858/) (2023)
- [Modeling Motivational Interviewing Strategies On An Online Peer-to-Peer Counseling Platform](https://mlsanity.com/paper/2211.05182/) (2022)
- [Practical Galaxy Morphology Tools from Deep Supervised Representation Learning](https://mlsanity.com/paper/2110.12735/) (2021)
- [ArmanEmo: A Persian Dataset for Text-based Emotion Detection](https://mlsanity.com/paper/2207.11808/) (2022)
- [Financial Sentiment Analysis on News and Reports Using Large Language Models and FinBERT](https://mlsanity.com/paper/2410.01987/) (2024)
- [Large Language Models for Propaganda Detection](https://mlsanity.com/paper/2310.06422/) (2023)
- [Team Phoenix at WASSA 2021: Emotion Analysis on News Stories with Pre-Trained Language Models](https://mlsanity.com/paper/2103.06057/) (2021)
- [Harnessing Artificial Intelligence to Combat Online Hate: Exploring the Challenges and Opportunities of Large Language Models in Hate Speech Detection](https://mlsanity.com/paper/2403.08035/) (2024)
- [Evaluating multiple models using labeled and unlabeled data](https://mlsanity.com/paper/2501.11866/) (2025)
- [An Empirical Study of IoT Security Aspects at Sentence-Level in Developer Textual Discussions](https://mlsanity.com/paper/2206.03079/) (2022)
- [Deep Reinforcement Learning Framework for Thoracic Diseases Classification via Prior Knowledge Guidance](https://mlsanity.com/paper/2306.01232/) (2023)
- [Stress index strategy enhanced with financial news sentiment analysis for the equity markets](https://mlsanity.com/paper/2404.00012/) (2024)
- [Zero-shot prompt-based classification: topic labeling in times of foundation models in German Tweets](https://mlsanity.com/paper/2406.18239/) (2024)
- [Evaluating ChatGPT on Nuclear Domain-Specific Data](https://mlsanity.com/paper/2409.00090/) (2024)
- [Classification of Safety Events at Nuclear Sites using Large Language Models](https://mlsanity.com/paper/2409.00091/) (2024)
- [JARVix at SemEval-2022 Task 2: It Takes One to Know One? Idiomaticity Detection using Zero and One-Shot Learning](https://mlsanity.com/paper/2202.02394/) (2022)
- [SASICM A Multi-Task Benchmark For Subtext Recognition](https://mlsanity.com/paper/2106.06944/) (2021)
- [Providing Insights for Open-Response Surveys via End-to-End Context-Aware Clustering](https://mlsanity.com/paper/2203.01294/) (2022)
- [Analysing the Greek Parliament Records with Emotion Classification](https://mlsanity.com/paper/2205.12012/) (2022)
- [Characterizing the Emotion Carriers of COVID-19 Misinformation and Their Impact on Vaccination Outcomes in India and the United States](https://mlsanity.com/paper/2306.13954/) (2023)
- [Integrating Text and Time-Series into (Large) Language Models to Predict Medical Outcomes](https://mlsanity.com/paper/2509.13696/) (2025)

## Abstract (arXiv)

Multi-emotion sentiment classification is a natural language processing (NLP) problem with valuable use cases on real-world data. We demonstrate that large-scale unsupervised language modeling combined with finetuning offers a practical solution to this task on difficult datasets, including those with label class imbalance and domain-specific context. By training an attention-based Transformer network (Vaswani et al. 2017) on 40GB of text (Amazon reviews) (McAuley et al. 2015) and fine-tuning on the training set, our model achieves a 0.69 F1 score on the SemEval Task 1:E-c multi-dimensional emotion classification problem (Mohammad et al. 2018), based on the Plutchik wheel of emotions (Plutchik 1979). These results are competitive with state of the art models, including strong F1 scores on difficult (emotion) categories such as Fear (0.73), Disgust (0.77) and Anger (0.78), as well as competitive results on rare categories such as Anticipation (0.42) and Surprise (0.37). Furthermore, we demonstrate our application on a real world text classification task. We create a narrowly collected text dataset of real tweets on several topics, and show that our finetuned model outperforms general purpose commercially available APIs for sentiment and multidimensional emotion classification on this dataset by a significant margin. We also perform a variety of additional studies, investigating properties of deep learning architectures, datasets and algorithms for achieving practical multidimensional sentiment classification. Overall, we find that unsupervised language modeling and finetuning is a simple framework for achieving high quality results on real-world sentiment classification.