# Assessing the societal influence of academic research with ChatGPT: Impact case study evaluations

arXiv: https://arxiv.org/abs/2410.19948 · HTML: https://mlsanity.com/paper/2410.19948/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 61 references that are in the map.
- First posted on arXiv 25 Oct 2024.

## Builds on (most cited first, 6 of 6)

- [Can ChatGPT evaluate research quality?](https://mlsanity.com/paper/2402.05519/) (2024)
- [Overton -- A bibliometric database of policy document citations](https://mlsanity.com/paper/2201.07643/) (2022)
- [Evaluating Research Quality with Large Language Models: An Analysis of ChatGPT's Effectiveness with Different Settings and Inputs](https://mlsanity.com/paper/2408.06752/) (2024)
- [In which fields can ChatGPT detect journal article quality? An evaluation of REF2021 results](https://mlsanity.com/paper/2409.16695/) (2024)
- [Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK Research Excellence Framework (REF)](https://mlsanity.com/paper/1807.03977/) (2018)
- [Do altmetric scores reflect article quality? Evidence from the UK Research Excellence Framework 2021](https://mlsanity.com/paper/2212.07811/) (2022)

## Built on by (most cited first, 3 of 3)

- [Evaluating the quality of published medical research with ChatGPT](https://mlsanity.com/paper/2411.01952/) (2024)
- [Can Smaller Large Language Models Evaluate Research Quality?](https://mlsanity.com/paper/2508.07196/) (2025)
- [Large language models for post-publication research evaluation: Evidence from expert recommendations and citation indicators](https://mlsanity.com/paper/2604.16387/) (2026)

## Abstract (arXiv)

Academics and departments are sometimes judged by how their research has benefitted society. For example, the UK Research Excellence Framework (REF) assesses Impact Case Studies (ICS), which are five-page evidence-based claims of societal impacts. This study investigates whether ChatGPT can evaluate societal impact claims and therefore potentially support expert human assessors. For this, various parts of 6,220 public ICS from REF2021 were fed to ChatGPT 4o-mini along with the REF2021 evaluation guidelines, comparing the results with published departmental average ICS scores. The results suggest that the optimal strategy for high correlations with expert scores is to input the title and summary of an ICS but not the remaining text, and to modify the original REF guidelines to encourage a stricter evaluation. The scores generated by this approach correlated positively with departmental average scores in all 34 Units of Assessment (UoAs), with values between 0.18 (Economics and Econometrics) and 0.56 (Psychology, Psychiatry and Neuroscience). At the departmental level, the corresponding correlations were higher, reaching 0.71 for Sport and Exercise Sciences, Leisure and Tourism. Thus, ChatGPT-based ICS evaluations are simple and viable to support or cross-check expert judgments, although their value varies substantially between fields.