# Can ChatGPT evaluate research quality?

arXiv: https://arxiv.org/abs/2402.05519 · HTML: https://mlsanity.com/paper/2402.05519/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 33 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Builds on (most cited first, 4 of 4)

- [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://mlsanity.com/paper/2302.10205/) (2023)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [Predicting article quality scores with machine learning: The UK Research Excellence Framework](https://mlsanity.com/paper/2212.05415/) (2022)

## Built on by (most cited first, 17 of 17)

- [Large language models for automated scholarly paper review: A survey](https://mlsanity.com/paper/2501.10326/) (2025)
- [A review on the novelty measurements of academic papers](https://mlsanity.com/paper/2501.17456/) (2025)
- [Academic journals' AI policies fail to curb the surge in AI-assisted academic writing](https://mlsanity.com/paper/2512.06705/) (2025)
- [Evaluating the Predictive Capacity of ChatGPT for Academic Peer Review Outcomes Across Multiple Platforms](https://mlsanity.com/paper/2411.09763/) (2024)
- [SC4ANM: Identifying Optimal Section Combinations for Automated Novelty Prediction in Academic Papers](https://mlsanity.com/paper/2505.16330/) (2025)
- [Evaluating Research Quality with Large Language Models: An Analysis of ChatGPT's Effectiveness with Different Settings and Inputs](https://mlsanity.com/paper/2408.06752/) (2024)
- [Advancing the Scientific Method with Large Language Models: From Hypothesis to Discovery](https://mlsanity.com/paper/2505.16477/) (2025)
- [Assessing the societal influence of academic research with ChatGPT: Impact case study evaluations](https://mlsanity.com/paper/2410.19948/) (2024)
- [Evaluating the quality of published medical research with ChatGPT](https://mlsanity.com/paper/2411.01952/) (2024)
- [Comparative Evaluation of ChatGPT and DeepSeek Across Key NLP Tasks: Strengths, Weaknesses, and Domain-Specific Performance](https://mlsanity.com/paper/2506.18501/) (2025)
- [Generative AI and the future of scientometrics: current topics and future questions](https://mlsanity.com/paper/2507.00783/) (2025)
- [Can Small and Reasoning Large Language Models Score Journal Articles for Research Quality and Do Averaging and Few-shot Help?](https://mlsanity.com/paper/2510.22389/) (2025)
- [EchoReview: Learning Peer Review from the Echoes of Scientific Citations](https://mlsanity.com/paper/2602.00733/) (2026)
- [Generics in science communication: Misaligned interpretations across laypeople, scientists, and large language models](https://mlsanity.com/paper/2602.06190/) (2026)
- [Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI](https://mlsanity.com/paper/2604.19578/) (2026)
- [Intelligence Is Not the Bottleneck: Validating an LLM First-Pass Manuscript Score Against Peer-Review Outcomes](https://mlsanity.com/paper/2606.15887/) (2026)
- [Beauty is in the AI of the beholder: MLLMs systematically overrate facial attractiveness](https://mlsanity.com/paper/2609.02512/) (2026)

## Abstract (arXiv)

Purpose: Assess whether ChatGPT 4.0 is accurate enough to perform research evaluations on journal articles to automate this time-consuming task. Design/methodology/approach: Test the extent to which ChatGPT-4 can assess the quality of journal articles using a case study of the published scoring guidelines of the UK Research Excellence Framework (REF) 2021 to create a research evaluation ChatGPT. This was applied to 51 of my own articles and compared against my own quality judgements. Findings: ChatGPT-4 can produce plausible document summaries and quality evaluation rationales that match the REF criteria. Its overall scores have weak correlations with my self-evaluation scores of the same documents (averaging r=0.281 over 15 iterations, with 8 being statistically significantly different from 0). In contrast, the average scores from the 15 iterations produced a statistically significant positive correlation of 0.509. Thus, averaging scores from multiple ChatGPT-4 rounds seems more effective than individual scores. The positive correlation may be due to ChatGPT being able to extract the author's significance, rigour, and originality claims from inside each paper. If my weakest articles are removed, then the correlation with average scores (r=0.200) falls below statistical significance, suggesting that ChatGPT struggles to make fine-grained evaluations. Research limitations: The data is self-evaluations of a convenience sample of articles from one academic in one field. Practical implications: Overall, ChatGPT does not yet seem to be accurate enough to be trusted for any formal or informal research quality evaluation tasks. Research evaluators, including journal editors, should therefore take steps to control its use. Originality/value: This is the first published attempt at post-publication expert review accuracy testing for ChatGPT.