# Which Model Shall I Choose? Cost/Quality Trade-offs for Text Classification Tasks

arXiv: https://arxiv.org/abs/2301.07006 · HTML: https://mlsanity.com/paper/2301.07006/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 16 references that are in the map.
- First posted on arXiv 17 Jan 2023.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Deep Learning Based Text Classification: A Comprehensive Review](https://mlsanity.com/paper/2004.03705/) (2020)
- [The Cost of Training NLP Models: A Concise Overview](https://mlsanity.com/paper/2004.08900/) (2020)
- [A Survey on Text Classification: From Shallow to Deep Learning](https://mlsanity.com/paper/2008.00364/) (2020)
- [How to Train BERT with an Academic Budget](https://mlsanity.com/paper/2104.07705/) (2021)
- [Pre-train or Annotate? Domain Adaptation with a Constrained Budget](https://mlsanity.com/paper/2109.04711/) (2021)

## Built on by (most cited first, 4 of 4)

- [Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling](https://mlsanity.com/paper/2308.06077/) (2023)
- [OptLLM: Optimal Assignment of Queries to Large Language Models](https://mlsanity.com/paper/2405.15130/) (2024)
- [JurEE not Judges: safeguarding llm interactions with small, specialised Encoder Ensembles](https://mlsanity.com/paper/2410.08442/) (2024)
- [EventChat: Implementation and user-centric evaluation of a large language model-driven conversational recommender system for exploring leisure events in an SME context](https://mlsanity.com/paper/2407.04472/) (2024)

## Abstract (arXiv)

Industry practitioners always face the problem of choosing the appropriate model for deployment under different considerations, such as to maximize a metric that is crucial for production, or to reduce the total cost given financial concerns. In this work, we focus on the text classification task and present a quantitative analysis for this challenge. Using classification accuracy as the main metric, we evaluate the classifiers' performances for a variety of models, including large language models, along with their associated costs, including the annotation cost, training (fine-tuning) cost, and inference cost. We then discuss the model choices for situations like having a large number of samples needed for inference. We hope our work will help people better understand the cost/quality trade-offs for the text classification task.