# Predicting article quality scores with machine learning: The UK Research Excellence Framework

arXiv: https://arxiv.org/abs/2212.05415 · HTML: https://mlsanity.com/paper/2212.05415/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 77 references that are in the map.
- First posted on arXiv 11 Dec 2022.

## Builds on (most cited first, 15 of 15)

- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [Can We Automate Scientific Reviewing?](https://mlsanity.com/paper/2102.00176/) (2021)
- [Predicting citation counts based on deep neural network learning techniques](https://mlsanity.com/paper/1809.04365/) (2018)
- [International Research Collaboration: Novelty, Conventionality, and Atypicality in Knowledge Recombination](https://mlsanity.com/paper/1804.09070/) (2018)
- [Early Indicators of Scientific Impact: Predicting Citations with Altmetrics](https://mlsanity.com/paper/2012.13599/) (2020)
- [Three practical field normalised alternative indicator formulae for research evaluation](https://mlsanity.com/paper/1612.01431/) (2016)
- [Does the use of open, non-anonymous peer review in scholarly publishing introduce bias? Evidence from the F1000 post-publication open peer review publishing model](https://mlsanity.com/paper/1911.03379/) (2019)
- [Bibliographic Analysis on Research Publications using Authors, Categorical Labels and the Citation Network](https://mlsanity.com/paper/1609.06532/) (2016)
- [Is research with qualitative data more prevalent and impactful now? Interviews, case studies, focus groups and ethnographies](https://mlsanity.com/paper/2104.11943/) (2021)
- [Geometric journal impact factors correcting for individual highly cited articles](https://mlsanity.com/paper/1510.08874/) (2015)
- [National, disciplinary and temporal variations in the extent to which articles with more authors have more impact: Evidence from a geometric field normalised citation indicator](https://mlsanity.com/paper/1511.07182/) (2015)
- [Systematic analysis of agreement between metrics and peer review in the UK REF](https://mlsanity.com/paper/1808.03491/) (2018)
- [Are the authors of highly cited articles also the most productive ones?](https://mlsanity.com/paper/1810.12842/) (2018)
- [Statistical relationships between corresponding authorship, international co-authorship and citation impact of national research systems](https://mlsanity.com/paper/2006.03366/) (2020)

## Built on by (most cited first, 3 of 3)

- [Can ChatGPT evaluate research quality?](https://mlsanity.com/paper/2402.05519/) (2024)
- [LLM-based feature generation from text for interpretable machine learning](https://mlsanity.com/paper/2409.07132/) (2024)
- [Evaluating Research Quality with Large Language Models: An Analysis of ChatGPT's Effectiveness with Different Settings and Inputs](https://mlsanity.com/paper/2408.06752/) (2024)

## Abstract (arXiv)

National research evaluation initiatives and incentive schemes have previously chosen between simplistic quantitative indicators and time-consuming peer review, sometimes supported by bibliometrics. Here we assess whether artificial intelligence (AI) could provide a third alternative, estimating article quality using more multiple bibliometric and metadata inputs. We investigated this using provisional three-level REF2021 peer review scores for 84,966 articles submitted to the UK Research Excellence Framework 2021, matching a Scopus record 2014-18 and with a substantial abstract. We found that accuracy is highest in the medical and physical sciences Units of Assessment (UoAs) and economics, reaching 42% above the baseline (72% overall) in the best case. This is based on 1000 bibliometric inputs and half of the articles used for training in each UoA. Prediction accuracies above the baseline for the social science, mathematics, engineering, arts, and humanities UoAs were much lower or close to zero. The Random Forest Classifier (standard or ordinal) and Extreme Gradient Boosting Classifier algorithms performed best from the 32 tested. Accuracy was lower if UoAs were merged or replaced by Scopus broad categories. We increased accuracy with an active learning strategy and by selecting articles with higher prediction probabilities, as estimated by the algorithms, but this substantially reduced the number of scores predicted.