# Selecting Between BERT and GPT for Text Classification in Political Science Research

arXiv: https://arxiv.org/abs/2411.05050 · HTML: https://mlsanity.com/paper/2411.05050/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 67 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Builds on (most cited first, 3 of 3)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT](https://mlsanity.com/paper/2302.10198/) (2023)
- [A Tutorial on the Pretrain-Finetune Paradigm for Natural Language Processing](https://mlsanity.com/paper/2403.02504/) (2024)

## Built on by (most cited first, 2 of 2)

- [Unpacking Robustness in Inflectional Languages: Adversarial Evaluation and Mechanistic Insights](https://mlsanity.com/paper/2505.07856/) (2025)
- [The Sound of Populism: Distinct Linguistic Features Across Populist Variants](https://mlsanity.com/paper/2505.07874/) (2025)

## Abstract (arXiv)

Political scientists often grapple with data scarcity in text classification. Recently, fine-tuned BERT models and their variants have gained traction as effective solutions to address this issue. In this study, we investigate the potential of GPT-based models combined with prompt engineering as a viable alternative. We conduct a series of experiments across various classification tasks, differing in the number of classes and complexity, to evaluate the effectiveness of BERT-based versus GPT-based models in low-data scenarios. Our findings indicate that while zero-shot and few-shot learning with GPT models provide reasonable performance and are well-suited for early-stage research exploration, they generally fall short - or, at best, match - the performance of BERT fine-tuning, particularly as the training set reaches a substantial size (e.g., 1,000 samples). We conclude by comparing these approaches in terms of performance, ease of use, and cost, providing practical guidance for researchers facing data limitations. Our results are particularly relevant for those engaged in quantitative text analysis in low-resource settings or with limited labeled data.