# Can Small and Reasoning Large Language Models Score Journal Articles for Research Quality and Do Averaging and Few-shot Help?

arXiv: https://arxiv.org/abs/2510.22389 · HTML: https://mlsanity.com/paper/2510.22389/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 25 Oct 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 19 of 19)

- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws](https://mlsanity.com/paper/2401.00448/) (2023)
- [Multi-Step Reasoning with Large Language Models, a Survey](https://mlsanity.com/paper/2407.11511/) (2024)
- [Task Contamination: Language Models May Not Be Few-Shot Anymore](https://mlsanity.com/paper/2312.16337/) (2023)
- [Understanding Reasoning in Thinking Language Models via Steering Vectors](https://mlsanity.com/paper/2506.18167/) (2025)
- [A Comparative Study on Reasoning Patterns of OpenAI's o1 Model](https://mlsanity.com/paper/2410.13639/) (2024)
- [Emergent Abilities in Large Language Models: A Survey](https://mlsanity.com/paper/2503.05788/) (2025)
- [Can ChatGPT evaluate research quality?](https://mlsanity.com/paper/2402.05519/) (2024)
- [Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approach](https://mlsanity.com/paper/2503.10573/) (2025)
- [Evaluating Research Quality with Large Language Models: An Analysis of ChatGPT's Effectiveness with Different Settings and Inputs](https://mlsanity.com/paper/2408.06752/) (2024)
- [In which fields can ChatGPT detect journal article quality? An evaluation of REF2021 results](https://mlsanity.com/paper/2409.16695/) (2024)
- [Research quality evaluation by AI in the era of Large Language Models: Advantages, disadvantages, and systemic effects](https://mlsanity.com/paper/2506.07748/) (2025)
- [Evaluating the quality of published medical research with ChatGPT](https://mlsanity.com/paper/2411.01952/) (2024)
- [Research evaluation with ChatGPT: Is it age, country, length, or field biased?](https://mlsanity.com/paper/2411.09768/) (2024)
- [Implicit and Explicit Research Quality Score Probabilities from ChatGPT](https://mlsanity.com/paper/2506.13525/) (2025)
- [Peer review and citation data in predicting university rankings, a large-scale analysis](https://mlsanity.com/paper/1805.08529/) (2018)
- [Comparative Evaluation of ChatGPT and DeepSeek Across Key NLP Tasks: Strengths, Weaknesses, and Domain-Specific Performance](https://mlsanity.com/paper/2506.18501/) (2025)
- [Can Smaller Large Language Models Evaluate Research Quality?](https://mlsanity.com/paper/2508.07196/) (2025)

## Built on by (most cited first, 1 of 1)

- [Designing large language model prompts to extract scores from messy text: A shared dataset and challenge](https://mlsanity.com/paper/2601.18271/) (2026)

## Abstract (arXiv)

Previous research has shown that journal article quality ratings from the cloud based Large Language Model (LLM) families ChatGPT and Gemini and the medium sized open weights LLM Gemma3 27b correlate moderately with expert research quality scores. This article assesses whether other medium sized LLMs, smaller LLMs, and reasoning models have similar abilities. This is tested with Gemma3 variants, Llama4 Scout, Qwen3, Magistral Small and DeepSeek R1 on a dataset of 2,780 medical, health and life science papers in 6 fields, with two different gold standards, one novel. Few-shot and score averaging approaches are also evaluated. The results suggest that medium-sized LLMs have similar performance to ChatGPT 4o-mini and Gemini 2.0 Flash, but that 1b parameters may often, and 4b sometimes, be too few. Reasoning models did not have a clear advantage. Moreover, averaging scores from multiple identical queries seems to be a universally successful strategy, and there is weak evidence that few-shot prompts (four examples) tend to help. Overall, the results show, for the first time, that smaller LLMs >4b have a substantial capability to rate journal articles for research quality, especially if score averaging is used, but that reasoning does not give an advantage for this task; it is therefore not recommended because it is slow. The use of LLMs to support research evaluation is now more credible since multiple variants have a similar ability, including many that can be deployed offline in a secure environment without substantial computing resources.