# Large Language Models as Evaluators for Scientific Synthesis

arXiv: https://arxiv.org/abs/2407.02977 · HTML: https://mlsanity.com/paper/2407.02977/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 22 references that are in the map.
- First posted on arXiv 3 Jul 2024.

## Builds on (most cited first, 11 of 11)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [Neural Text Summarization: A Critical Evaluation](https://mlsanity.com/paper/1908.08960/) (2019)
- [Benchmarking Foundation Models with Language-Model-as-an-Examiner](https://mlsanity.com/paper/2306.04181/) (2023)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- [SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization](https://mlsanity.com/paper/2005.03724/) (2020)
- [Revisiting Summarization Evaluation for Scientific Articles](https://mlsanity.com/paper/1604.00400/) (2016)
- [CORE-GPT: Combining Open Access research and large language models for credible, trustworthy question answering](https://mlsanity.com/paper/2307.04683/) (2023)

## Built on by (most cited first, 5 of 5)

- [YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering](https://mlsanity.com/paper/2505.14279/) (2025)
- [LLMs4Synthesis: Leveraging Large Language Models for Scientific Synthesis](https://mlsanity.com/paper/2409.18812/) (2024)
- [One world, one opinion? The superstar effect in LLM responses](https://mlsanity.com/paper/2412.10281/) (2024)
- [Multi-Agent Retrieval-Augmented Framework for Evidence-Based Counterspeech Against Health Misinformation](https://mlsanity.com/paper/2507.07307/) (2025)
- [Impact of large language models on peer review opinions from a fine-grained perspective: Evidence from top conference proceedings in AI](https://mlsanity.com/paper/2604.19578/) (2026)

## Abstract (arXiv)

Our study explores how well the state-of-the-art Large Language Models (LLMs), like GPT-4 and Mistral, can assess the quality of scientific summaries or, more fittingly, scientific syntheses, comparing their evaluations to those of human annotators. We used a dataset of 100 research questions and their syntheses made by GPT-4 from abstracts of five related papers, checked against human quality ratings. The study evaluates both the closed-source GPT-4 and the open-source Mistral model's ability to rate these summaries and provide reasons for their judgments. Preliminary results show that LLMs can offer logical explanations that somewhat match the quality ratings, yet a deeper statistical analysis shows a weak correlation between LLM and human ratings, suggesting the potential and current limitations of LLMs in scientific synthesis evaluation.