TextClass Benchmark: A Continuous Elo Rating of LLMs in Social Sciences
Not on the map. Why?
Abstract (arXiv)
The TextClass Benchmark project is an ongoing, continuous benchmarking process that aims to provide a comprehensive, fair, and dynamic evaluation of LLMs and transformers for text classification tasks. This evaluation spans various domains and languages in social sciences disciplines engaged in NLP and text-as-data approach. The leaderboards present performance metrics and relative ranking using a tailored Elo rating system. With each leaderboard cycle, novel models are added, fixed test sets can be replaced for unseen, equivalent data to test generalisation power, ratings are updated, and a Meta-Elo leaderboard combines and weights domain-specific leaderboards. This article presents the rationale and motivation behind the project, explains the Elo rating system in detail, and estimates Meta-Elo across different classification tasks in social science disciplines. We also present a snapshot of the first cycle of classification tasks on incivility data in Chinese, English, German and Russian. This ongoing benchmarking process includes not only additional languages such as Arabic, Hindi, and Spanish but also a classification of policy agenda topics, misinformation, among others.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 6 of its 20 references that are in the map. First posted on arXiv 30 Nov 2024.
Builds on
6 of 20 references.
References · 6
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
- 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
- 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
- 2023Evaluation is all you need. Prompting Generative Large Language Models for Annotation Tasks in the Social Sciences. A Primer using Open Modelscited by 12
- 2024Benchmarking LLMs in Political Content Text-Annotation: Proof-of-Concept with Toxicity and Incivility Datacited by 4
- 2024rollama: An R package for using generative large language models through Ollamacited by 3
Built on by
Citing papers per year: 2025: 1.
1 paper.