mlsanityData to 14 Oct 2026

Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs?

arXiv 2505.18215 ↗First posted 23 May 2025cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

The rapid adoption of LLMs has overshadowed the potential advantages of traditional BERT-like models in text classification. This study challenges the prevailing "LLM-centric" trend by systematically comparing three category methods, i.e., BERT-like models fine-tuning, LLM internal state utilization, and zero-shot inference across six high-difficulty datasets. Our findings reveal that BERT-like models often outperform LLMs. We further categorize datasets into three types, perform PCA and probing experiments, and identify task-specific model strengths: BERT-like models excel in pattern-driven tasks, while LLMs dominate those requiring deep semantics or world knowledge. Based on this, we propose TaMAS, a fine-grained task selection strategy, advocating for a nuanced, task-driven approach over a one-size-fits-all reliance on LLMs.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 15 of its 35 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

15 of 35 references.

References · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  5. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  6. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  7. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  8. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 260
  9. 2019ERNIE 2.0: A Continual Pre-training Framework for Language Understandingcited by 231
  10. 2024Pushing The Limit of LLM Capacity for Text Classificationcited by 28
  11. 2024From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasetscited by 25
  12. 2025CoheMark: A Novel Sentence-Level Watermark for Enhanced Text Qualitycited by 19
3 more
  1. 2024Exploring Large-Scale Language Models to Evaluate EEG-Based Multimodal Data for Mental Healthcited by 13
  2. 2025Text Classification in the LLM Era -- Where do we stand?cited by 10
  3. 2025RePPL: Recalibrating Perplexity by Uncertainty in Semantic Propagation and Language Generation for Explainable QA Hallucination Detectioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.

  1. 2025VLA-Mark: A cross modal watermark for large vision-language alignment modelcited by 12
  2. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steeringcited by 8
  3. 2026BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluationcited by 4
  4. 2025GENUINE: Graph Enhanced Multi-level Uncertainty Estimation for Large Language Modelscited by 3
  5. 2026Distilling the Thought, Watermarking the Answer: A Principle Semantic Guided Watermark for Large Reasoning Modelscited by 3
  6. 2026A Visual Semantic Adaptive Watermark grounded by Prefix-Tuning for Large Vision-Language Modelcited by 3
  7. 2026Sell More, Play Less: Benchmarking LLM Realistic Selling Skillcited by 3
  8. 2026The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLPcited by 0
  9. 2026UltraBERT: Beyond the Limit of BERT Pre-Trainingcited by 0
  10. 2026Swiss4Arg: A Parallel Dataset for Multilingual Argument Mining across Swiss National Languagescited by 0