# Pushing The Limit of LLM Capacity for Text Classification

arXiv: https://arxiv.org/abs/2402.07470 · HTML: https://mlsanity.com/paper/2402.07470/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 53 references that are in the map.
- First posted on arXiv 12 Feb 2024.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/1809.05679/) (2018)
- [ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2107.02137/) (2021)
- [PTR: Prompt Tuning with Rules for Text Classification](https://mlsanity.com/paper/2105.11259/) (2021)
- [Recurrent Neural Network for Text Classification with Multi-Task Learning](https://mlsanity.com/paper/1605.05101/) (2016)
- [Text Classification via Large Language Models](https://mlsanity.com/paper/2305.08377/) (2023)
- [HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs](https://mlsanity.com/paper/2311.09774/) (2023)
- [BertGCN: Transductive Text Classification by Combining GCN and BERT](https://mlsanity.com/paper/2105.05727/) (2021)
- [Explaining the Success of AdaBoost and Random Forests as Interpolating Classifiers](https://mlsanity.com/paper/1504.07676/) (2015)
- [PromptBoosting: Black-Box Text Classification with Ten Forward Passes](https://mlsanity.com/paper/2212.09257/) (2022)

## Built on by (most cited first, 28 of 28)

- [A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directions](https://mlsanity.com/paper/2412.19823/) (2024)
- [TaxoAdapt: Aligning LLM-Based Multidimensional Taxonomy Construction to Evolving Research Corpora](https://mlsanity.com/paper/2506.10737/) (2025)
- [SarcasmBench: Towards Evaluating Large Language Models on Sarcasm Understanding](https://mlsanity.com/paper/2408.11319/) (2024)
- [Improving LLM Reasoning through Interpretable Role-Playing Steering](https://mlsanity.com/paper/2506.07335/) (2025)
- [From Prompts to Power: Measuring the Energy Footprint of LLM Inference](https://mlsanity.com/paper/2511.05597/) (2025)
- [RAMIE: Retrieval-Augmented Multi-task Information Extraction with Large Language Models on Dietary Supplements](https://mlsanity.com/paper/2411.15700/) (2024)
- [Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs?](https://mlsanity.com/paper/2505.18215/) (2025)
- [Large Language Models for Subjective Language Understanding: A Survey](https://mlsanity.com/paper/2508.07959/) (2025)
- [Examining the Prevalence and Dynamics of AI-Generated Media in Art Subreddits](https://mlsanity.com/paper/2410.07302/) (2024)
- [LLMs' Classification Performance is Overclaimed](https://mlsanity.com/paper/2406.16203/) (2024)
- [CluCERT: Certifying LLM Robustness via Clustering-Guided Denoising Smoothing](https://mlsanity.com/paper/2512.08967/) (2025)
- [From An LLM Swarm To A PDDL-Empowered HIVE: Planning Self-Executed Instructions In A Multi-Modal Jungle](https://mlsanity.com/paper/2412.12839/) (2024)
- [Performance Characterization of Expert Router for Scalable LLM Inference](https://mlsanity.com/paper/2404.15153/) (2024)
- [Gen-AI for User Safety: A Survey](https://mlsanity.com/paper/2411.06606/) (2024)
- [Regulation of Language Models With Interpretability Will Likely Result In A Performance Trade-Off](https://mlsanity.com/paper/2412.12169/) (2024)
- [Data Quality Enhancement on the Basis of Diversity with Large Language Models for Text Classification: Uncovered, Difficult, and Noisy](https://mlsanity.com/paper/2412.06575/) (2024)
- [QUAD-LLM-MLTC: Large Language Models Ensemble Learning for Healthcare Text Multi-Label Classification](https://mlsanity.com/paper/2502.14189/) (2025)
- [Integrated ensemble of BERT- and features-based models for authorship attribution in Japanese literary works](https://mlsanity.com/paper/2504.08527/) (2025)
- [State of the Art in Text Classification for South Slavic Languages: Fine-Tuning or Prompting?](https://mlsanity.com/paper/2511.07989/) (2025)
- [From Description to Score: Can LLMs Quantify Vulnerabilities?](https://mlsanity.com/paper/2512.06781/) (2025)
- [Seventeenth-Century Spanish American Notary Records for Fine-Tuning Spanish Large Language Models](https://mlsanity.com/paper/2406.05812/) (2024)
- [A Transformer-based Autoregressive Decoder Architecture for Hierarchical Text Classification](https://mlsanity.com/paper/2501.13598/) (2025)
- [Improving Multi-turn Task Completion in Task-Oriented Dialog Systems via Prompt Chaining and Fine-Grained Feedback](https://mlsanity.com/paper/2502.13298/) (2025)
- [Comparative Study on the Discourse Meaning of Chinese and English Media in the Paris Olympics Based on LDA Topic Modeling Technology and LLM Prompt Engineering](https://mlsanity.com/paper/2504.18106/) (2025)
- [A Comparative Study of Task Adaptation Techniques of Large Language Models for Identifying Sustainable Development Goals](https://mlsanity.com/paper/2506.15208/) (2025)
- [Text2Graph: Combining Lightweight LLMs and GNNs for Efficient Text Classification in Label-Scarce Scenarios](https://mlsanity.com/paper/2512.10061/) (2025)
- [PerSoMed: A Large-Scale Balanced Dataset for Persian Social Media Text Classification](https://mlsanity.com/paper/2602.19333/) (2026)
- [ProbPlug: A Plugin Uncertainty Network for Reliable Confidence in LLM Binary Classification](https://mlsanity.com/paper/2609.10122/) (2026)

## Abstract (arXiv)

The value of text classification's future research has encountered challenges and uncertainties, due to the extraordinary efficacy demonstrated by large language models (LLMs) across numerous downstream NLP tasks. In this era of open-ended language modeling, where task boundaries are gradually fading, an urgent question emerges: have we made significant advances in text classification under the full benefit of LLMs? To answer this question, we propose RGPT, an adaptive boosting framework tailored to produce a specialized text classification LLM by recurrently ensembling a pool of strong base learners. The base learners are constructed by adaptively adjusting the distribution of training samples and iteratively fine-tuning LLMs with them. Such base learners are then ensembled to be a specialized text classification LLM, by recurrently incorporating the historical predictions from the previous learners. Through a comprehensive empirical comparison, we show that RGPT significantly outperforms 8 SOTA PLMs and 7 SOTA LLMs on four benchmarks by 1.36% on average. Further evaluation experiments show a clear surpassing of RGPT over human classification.