# Generative or Discriminative? Revisiting Text Classification in the Era of Transformers

arXiv: https://arxiv.org/abs/2506.12181 · HTML: https://mlsanity.com/paper/2506.12181/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 78 references that are in the map.
- First posted on arXiv 13 Jun 2025.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt](https://mlsanity.com/paper/2206.07137/) (2022)
- [DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models](https://mlsanity.com/paper/2211.15029/) (2022)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [Score-based Continuous-time Discrete Diffusion Models](https://mlsanity.com/paper/2211.16750/) (2022)
- [Impact of Pretraining Term Frequencies on Few-Shot Reasoning](https://mlsanity.com/paper/2202.07206/) (2022)
- [Understanding and Improving Information Transfer in Multi-Task Learning](https://mlsanity.com/paper/2005.00944/) (2020)
- [Certified Robustness to Text Adversarial Attacks by Randomized [MASK]](https://mlsanity.com/paper/2105.03743/) (2021)
- [Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspective](https://mlsanity.com/paper/2308.13985/) (2023)
- [Syntax-guided Controlled Generation of Paraphrases](https://mlsanity.com/paper/2005.08417/) (2020)
- [Are Generative Classifiers More Robust to Adversarial Attacks?](https://mlsanity.com/paper/1802.06552/) (2018)
- [When Does Optimizing a Proper Loss Yield Calibration?](https://mlsanity.com/paper/2305.18764/) (2023)
- [Text-CRS: A Generalized Certified Robustness Framework against Textual Adversarial Attacks](https://mlsanity.com/paper/2307.16630/) (2023)
- [Generative and Discriminative Text Classification with Recurrent Neural Networks](https://mlsanity.com/paper/1703.01898/) (2017)
- [How Robust are LLMs to In-Context Majority Label Bias?](https://mlsanity.com/paper/2312.16549/) (2023)

## Built on by (most cited first, 9 of 9)

- [LLM Safety From Within: Detecting Harmful Content with Internal Representations](https://mlsanity.com/paper/2604.18519/) (2026)
- [Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects](https://mlsanity.com/paper/2510.07890/) (2025)
- [Your VAR Model is Secretly an Efficient and Explainable Generative Classifier](https://mlsanity.com/paper/2510.12060/) (2025)
- [Fast-dLLM++: Fr'echet Profile Decoding for Faster Diffusion LLM Inference](https://mlsanity.com/paper/2606.02955/) (2026)
- [The Hidden Cost of Modeling P(X): Vulnerability to Membership Inference Attacks in Generative Text Classifiers](https://mlsanity.com/paper/2510.16122/) (2025)
- [Closing the Auto-Research Loop: An AI Co-Scientist for Production Search Ranking](https://mlsanity.com/paper/2603.22376/) (2026)
- [Unifying Acoustic Features and Text with Multimodal LLMs for Neurodegenerative Screening](https://mlsanity.com/paper/2606.14788/) (2026)
- [AEScorer: An Agentic Evidence-Grounded Framework for Graded Factuality Verification](https://mlsanity.com/paper/2601.03605/) (2026)
- [Revisiting Autoregressive Models for Generative Image Classification](https://mlsanity.com/paper/2603.19122/) (2026)

## Abstract (arXiv)

The comparison between discriminative and generative classifiers has intrigued researchers since Efron's seminal analysis of logistic regression versus discriminant analysis. While early theoretical work established that generative classifiers exhibit lower sample complexity but higher asymptotic error in simple linear settings, these trade-offs remain unexplored in the transformer era. We present the first comprehensive evaluation of modern generative and discriminative architectures - Auto-regressive modeling, Masked Language Modeling, Discrete Diffusion, and Encoders for text classification. Our study reveals that the classical 'two regimes' phenomenon manifests distinctly across different architectures and training paradigms. Beyond accuracy, we analyze sample efficiency, calibration, noise robustness, and ordinality across diverse scenarios. Our findings offer practical guidance for selecting the most suitable modeling approach based on real-world constraints such as latency and data limitations.