# Detecting Reference Errors in Scientific Literature with Large Language Models

arXiv: https://arxiv.org/abs/2411.06101 · HTML: https://mlsanity.com/paper/2411.06101/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 34 references that are in the map.
- First posted on arXiv 9 Nov 2024.

## Builds on (most cited first, 7 of 7)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [Scientific Claim Verification with VERT5ERINI](https://mlsanity.com/paper/2010.11930/) (2020)
- [A Paragraph-level Multi-task Learning Model for Scientific Fact-Verification](https://mlsanity.com/paper/2012.14500/) (2020)
- [How well do LLMs cite relevant medical references? An evaluation framework and analyses](https://mlsanity.com/paper/2402.02008/) (2024)

## Built on by (most cited first, 4 of 4)

- [Reviewing Scientific Papers for Critical Problems With Reasoning LLMs: Baseline Approaches and Automatic Evaluation](https://mlsanity.com/paper/2505.23824/) (2025)
- [BibAgent: An Agentic Framework for Traceable Miscitation Detection in Scientific Literature](https://mlsanity.com/paper/2601.16993/) (2026)
- [To Err Is Human: Systematic Quantification of Errors in Published AI Papers via LLM Analysis](https://mlsanity.com/paper/2512.05925/) (2025)
- [Detection and Interpretability Analysis of Quotation Errors by Large Language Models](https://mlsanity.com/paper/2606.08589/) (2026)

## Abstract (arXiv)

Reference errors, such as citation and quotation errors, are common in scientific papers. Such errors can result in the propagation of inaccurate information, but are difficult and time-consuming to detect, posing a significant challenge to scientific publishing. To support automatic detection of reference errors, this work evaluated the ability of large language models in OpenAI's GPT family to detect quotation errors. Specifically, we prepared an expert-annotated, general-domain dataset of statement-reference pairs from journal articles. Large language models were evaluated in different settings with varying amounts of reference information provided by retrieval augmentation. Our results showed that large language models are able to detect erroneous citations with limited context and without fine-tuning. This study contributes to the growing literature that seeks to utilize artificial intelligence to assist in the writing, reviewing, and publishing of scientific papers. Potential avenues for further improvements in this task are also discussed.