# In which fields can ChatGPT detect journal article quality? An evaluation of REF2021 results

arXiv: https://arxiv.org/abs/2409.16695 · HTML: https://mlsanity.com/paper/2409.16695/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2024.

## Built on by (most cited first, 5 of 5)

- [Assessing the societal influence of academic research with ChatGPT: Impact case study evaluations](https://mlsanity.com/paper/2410.19948/) (2024)
- [Evaluating the quality of published medical research with ChatGPT](https://mlsanity.com/paper/2411.01952/) (2024)
- [Can Small and Reasoning Large Language Models Score Journal Articles for Research Quality and Do Averaging and Few-shot Help?](https://mlsanity.com/paper/2510.22389/) (2025)
- [Designing large language model prompts to extract scores from messy text: A shared dataset and challenge](https://mlsanity.com/paper/2601.18271/) (2026)
- [Beyond Detection: Governing GenAI in Academic Peer Review as a Sociotechnical Challenge](https://mlsanity.com/paper/2603.20214/) (2026)

## Abstract (arXiv)

Time spent by academics on research quality assessment might be reduced if automated approaches can help. Whilst citation-based indicators have been extensively developed and evaluated for this, they have substantial limitations and Large Language Models (LLMs) like ChatGPT provide an alternative approach. This article assesses whether ChatGPT 4o-mini can be used to estimate the quality of journal articles across academia. It samples up to 200 articles from all 34 Units of Assessment (UoAs) in the UK's Research Excellence Framework (REF) 2021, comparing ChatGPT scores with departmental average scores. There was an almost universally positive Spearman correlation between ChatGPT scores and departmental averages, varying between 0.08 (Philosophy) and 0.78 (Psychology, Psychiatry and Neuroscience), except for Clinical Medicine (rho=-0.12). Although other explanations are possible, especially because REF score profiles are public, the results suggest that LLMs can provide reasonable research quality estimates in most areas of science, and particularly the physical and health sciences and engineering, even before citation data is available. Nevertheless, ChatGPT assessments seem to be more positive for most health and physical sciences than for other fields, a concern for multidisciplinary assessments, and the ChatGPT scores are only based on titles and abstracts, so cannot be research evaluations.