# Beyond the Hype: Embeddings vs. Prompting for Multiclass Classification Tasks

arXiv: https://arxiv.org/abs/2504.04277 · HTML: https://mlsanity.com/paper/2504.04277/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 23 references that are in the map.
- First posted on arXiv 5 Apr 2025.

## Builds on (most cited first, 5 of 5)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Pooling Methods in Deep Neural Networks, a Review](https://mlsanity.com/paper/2009.07485/) (2020)
- [Generative AI Text Classification using Ensemble LLM Approaches](https://mlsanity.com/paper/2309.07755/) (2023)
- [GRILLBot In Practice: Lessons and Tradeoffs Deploying Large Language Models for Adaptable Conversational Task Assistants](https://mlsanity.com/paper/2402.07647/) (2024)
- [Large Scale Generative AI Text Applied to Sports and Music](https://mlsanity.com/paper/2402.15514/) (2024)

## Abstract (arXiv)

Are traditional classification approaches irrelevant in this era of AI hype? We show that there are multiclass classification problems where predictive models holistically outperform LLM prompt-based frameworks. Given text and images from home-service project descriptions provided by Thumbtack customers, we build embeddings-based softmax models that predict the professional category (e.g., handyman, bathroom remodeling) associated with each problem description. We then compare against prompts that ask state-of-the-art LLM models to solve the same problem. We find that the embeddings approach outperforms the best LLM prompts in terms of accuracy, calibration, latency, and financial cost. In particular, the embeddings approach has 49.5% higher accuracy than the prompting approach, and its superiority is consistent across text-only, image-only, and text-image problem descriptions. Furthermore, it yields well-calibrated probabilities, which we later use as confidence signals to provide contextualized user experience during deployment. On the contrary, prompting scores are overly uninformative. Finally, the embeddings approach is 14 and 81 times faster than prompting in processing images and text respectively, while under realistic deployment assumptions, it can be up to 10 times cheaper. Based on these results, we deployed a variation of the embeddings approach, and through A/B testing we observed performance consistent with our offline analysis. Our study shows that for multiclass classification problems that can leverage proprietary datasets, an embeddings-based approach may yield unequivocally better results. Hence, scientists, practitioners, engineers, and business leaders can use our study to go beyond the hype and consider appropriate predictive models for their classification use cases.