# Evaluating Research Quality with Large Language Models: An Analysis of ChatGPT's Effectiveness with Different Settings and Inputs

arXiv: https://arxiv.org/abs/2408.06752 · HTML: https://mlsanity.com/paper/2408.06752/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 28 references that are in the map.
- First posted on arXiv 13 Aug 2024.

## Builds on (most cited first, 12 of 12)

- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews](https://mlsanity.com/paper/2403.07183/) (2024)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [Can ChatGPT evaluate research quality?](https://mlsanity.com/paper/2402.05519/) (2024)
- [Eliciting Informative Text Evaluations with Large Language Models](https://mlsanity.com/paper/2405.15077/) (2024)
- [ChatGPT vs Gemini vs LLaMA on Multilingual Sentiment Analysis](https://mlsanity.com/paper/2402.01715/) (2024)
- [Predicting article quality scores with machine learning: The UK Research Excellence Framework](https://mlsanity.com/paper/2212.05415/) (2022)

## Built on by (most cited first, 7 of 7)

- [Evaluating the Predictive Capacity of ChatGPT for Academic Peer Review Outcomes Across Multiple Platforms](https://mlsanity.com/paper/2411.09763/) (2024)
- [Assessing the societal influence of academic research with ChatGPT: Impact case study evaluations](https://mlsanity.com/paper/2410.19948/) (2024)
- [Evaluating the quality of published medical research with ChatGPT](https://mlsanity.com/paper/2411.01952/) (2024)
- [Can Smaller Large Language Models Evaluate Research Quality?](https://mlsanity.com/paper/2508.07196/) (2025)
- [Can Small and Reasoning Large Language Models Score Journal Articles for Research Quality and Do Averaging and Few-shot Help?](https://mlsanity.com/paper/2510.22389/) (2025)
- [LLEXICORP: End-user Explainability of Convolutional Neural Networks](https://mlsanity.com/paper/2511.02720/) (2025)
- [HexEval: An Evidence-Driven Hexagonal Framework for Multidimensional Scholar Assessment](https://mlsanity.com/paper/2608.10584/) (2026)

## Abstract (arXiv)

Evaluating the quality of academic journal articles is a time consuming but critical task for national research evaluation exercises, appointments and promotion. It is therefore important to investigate whether Large Language Models (LLMs) can play a role in this process. This article assesses which ChatGPT inputs (full text without tables, figures and references; title and abstract; title only) produce better quality score estimates, and the extent to which scores are affected by ChatGPT models and system prompts. The results show that the optimal input is the article title and abstract, with average ChatGPT scores based on these (30 iterations on a dataset of 51 papers) correlating at 0.67 with human scores, the highest ever reported. ChatGPT 4o is slightly better than 3.5-turbo (0.66), and 4o-mini (0.66). The results suggest that article full texts might confuse LLM research quality evaluations, even though complex system instructions for the task are more effective than simple ones. Thus, whilst abstracts contain insufficient information for a thorough assessment of rigour, they may contain strong pointers about originality and significance. Finally, linear regression can be used to convert the model scores into the human scale scores, which is 31% more accurate than guessing.