mlsanityData to 14 Oct 2026

Generative or Discriminative? Revisiting Text Classification in the Era of Transformers

arXiv 2506.12181 ↗First posted 13 Jun 2025cs.LG cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

The comparison between discriminative and generative classifiers has intrigued researchers since Efron's seminal analysis of logistic regression versus discriminant analysis. While early theoretical work established that generative classifiers exhibit lower sample complexity but higher asymptotic error in simple linear settings, these trade-offs remain unexplored in the transformer era. We present the first comprehensive evaluation of modern generative and discriminative architectures - Auto-regressive modeling, Masked Language Modeling, Discrete Diffusion, and Encoders for text classification. Our study reveals that the classical 'two regimes' phenomenon manifests distinctly across different architectures and training paradigms. Beyond accuracy, we analyze sample efficiency, calibration, noise robustness, and ordinality across diverse scenarios. Our findings offer practical guidance for selecting the most suitable modeling approach based on real-world constraints such as latency and data limitations.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 35 of its 78 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

35 of 78 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2017On Calibration of Modern Neural Networkscited by 4,960
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  9. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  10. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  11. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  12. 2020Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Imagescited by 745
23 more
  1. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
  2. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  3. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  4. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
  5. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  6. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  7. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 185
  8. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 169
  9. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  10. 2020Understanding and Improving Information Transfer in Multi-Task Learningcited by 114
  11. 2021Certified Robustness to Text Adversarial Attacks by Randomized [MASK]cited by 57
  12. 2023Revisiting Scalarization in Multi-Task Learning: A Theoretical Perspectivecited by 51
  13. 2020Syntax-guided Controlled Generation of Paraphrasescited by 47
  14. 2018Are Generative Classifiers More Robust to Adversarial Attacks?cited by 36
  15. 2023When Does Optimizing a Proper Loss Yield Calibration?cited by 36
  16. 2023Text-CRS: A Generalized Certified Robustness Framework against Textual Adversarial Attackscited by 34
  17. 2017Generative and Discriminative Text Classification with Recurrent Neural Networkscited by 31
  18. 2023How Robust are LLMs to In-Context Majority Label Bias?cited by 14
  19. 2023A Multimodal Prototypical Approach for Unsupervised Sound Classificationcited by 6
  20. 2023A Hybrid of Generative and Discriminative Models Based on the Gaussian-coupled Softmax Layercited by 4
  21. 2023Tackling Concept Shift in Text Classification using Entailment-style Modelingcited by 4
  22. 2021Discriminative and Generative Transformer-based Models For Situation Entity Classificationcited by 3
  23. 2024Exploring Ordinality in Text Classification: A Comparative Study of Explicit and Implicit Techniquescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 6.

9 papers.

  1. 2026LLM Safety From Within: Detecting Harmful Content with Internal Representationscited by 11
  2. 2025Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialectscited by 2
  3. 2025Your VAR Model is Secretly an Efficient and Explainable Generative Classifiercited by 2
  4. 2026Fast-dLLM++: Fr'echet Profile Decoding for Faster Diffusion LLM Inferencecited by 2
  5. 2025The Hidden Cost of Modeling P(X): Vulnerability to Membership Inference Attacks in Generative Text Classifierscited by 1
  6. 2026Closing the Auto-Research Loop: An AI Co-Scientist for Production Search Rankingcited by 1
  7. 2026Unifying Acoustic Features and Text with Multimodal LLMs for Neurodegenerative Screeningcited by 1
  8. 2026AEScorer: An Agentic Evidence-Grounded Framework for Graded Factuality Verificationcited by 0
  9. 2026Revisiting Autoregressive Models for Generative Image Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GenCLS++: Pushing the Boundaries of Generative Classification in LLMs Through Comprehensive SFT and RL Studies Across Diverse Datasets
  2. 2023SelfEval: Leveraging the discriminative nature of generative models for evaluation
  3. 2025Post-Training Quantization of Generative and Discriminative LSTM Text Classifiers: A Study of Calibration, Class Balance, and Robustness
  4. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  5. 2024Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance
  6. 2026Scaling and Distilling Text Embeddings for Better Diffusibility
  7. 2026Extending One-Step Image Generation from Class Labels to Text via Discriminative Text Representation
  8. 2026Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics