# CiteME: Can Language Models Accurately Cite Scientific Claims?

arXiv: https://arxiv.org/abs/2407.12861 · HTML: https://mlsanity.com/paper/2407.12861/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 107 references that are in the map.
- First posted on arXiv 10 Jul 2024.

## Builds on (most cited first, 30 of 60)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Measuring and Narrowing the Compositionality Gap in Language Models](https://mlsanity.com/paper/2210.03350/) (2022)
- [ChemCrow: Augmenting large-language models with chemistry tools](https://mlsanity.com/paper/2304.05376/) (2023)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [How Language Model Hallucinations Can Snowball](https://mlsanity.com/paper/2305.13534/) (2023)
- [tinyBenchmarks: evaluating LLMs with fewer examples](https://mlsanity.com/paper/2402.14992/) (2024)
- [Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence](https://mlsanity.com/paper/2103.08541/) (2021)
- [A Neural Conversational Model](https://mlsanity.com/paper/1506.05869/) (2015)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [PaperQA: Retrieval-Augmented Generative Agent for Scientific Research](https://mlsanity.com/paper/2312.07559/) (2023)
- [A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications](https://mlsanity.com/paper/1804.09635/) (2018)
- [Generating Fact Checking Explanations](https://mlsanity.com/paper/2004.05773/) (2020)
- [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://mlsanity.com/paper/2211.13308/) (2022)

## Built on by (most cited first, 30 of 48)

- [Agent Laboratory: Using LLM Agents as Research Assistants](https://mlsanity.com/paper/2501.04227/) (2025)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions](https://mlsanity.com/paper/2503.08979/) (2025)
- [AgentRxiv: Towards Collaborative Autonomous Research](https://mlsanity.com/paper/2503.18102/) (2025)
- [Automated Hypothesis Validation with Agentic Sequential Falsifications](https://mlsanity.com/paper/2502.09858/) (2025)
- [HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generation](https://mlsanity.com/paper/2504.11524/) (2025)
- [Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse](https://mlsanity.com/paper/2409.11242/) (2024)
- [Literature Meets Data: A Synergistic Approach to Hypothesis Generation](https://mlsanity.com/paper/2410.17309/) (2024)
- [IRIS: Interactive Research Ideation System for Accelerating Scientific Discovery](https://mlsanity.com/paper/2504.16728/) (2025)
- [ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence](https://mlsanity.com/paper/2605.26340/) (2026)
- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents](https://mlsanity.com/paper/2605.06635/) (2026)
- [AI for Auto-Research: Roadmap & User Guide](https://mlsanity.com/paper/2605.18661/) (2026)
- [How Deep Do Large Language Models Internalize Scientific Literature and Citation Practices?](https://mlsanity.com/paper/2504.02767/) (2025)
- [AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery](https://mlsanity.com/paper/2605.23204/) (2026)
- [HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputs](https://mlsanity.com/paper/2503.02003/) (2025)
- [LLM-REVal: Can We Trust LLM Reviewers Yet?](https://mlsanity.com/paper/2510.12367/) (2025)
- [BibAgent: An Agentic Framework for Traceable Miscitation Detection in Scientific Literature](https://mlsanity.com/paper/2601.16993/) (2026)
- [VeriCite: Towards Reliable Citations in Retrieval-Augmented Generation via Rigorous Verification](https://mlsanity.com/paper/2510.11394/) (2025)
- [LABBench2: An Improved Benchmark for AI Systems Performing Biology Research](https://mlsanity.com/paper/2604.09554/) (2026)
- [Can Language Models Falsify? Evaluating Algorithmic Reasoning with Counterexample Creation](https://mlsanity.com/paper/2502.19414/) (2025)
- [XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaboration](https://mlsanity.com/paper/2505.11336/) (2025)
- [FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes](https://mlsanity.com/paper/2506.03278/) (2025)
- [Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents](https://mlsanity.com/paper/2604.03173/) (2026)
- [Large Language Model Sourcing: A Survey](https://mlsanity.com/paper/2510.10161/) (2025)
- [ScholarGym: Benchmarking Large Language Model Capabilities in the Information-Gathering Stage of Deep Research](https://mlsanity.com/paper/2601.21654/) (2026)
- [A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations](https://mlsanity.com/paper/2603.25196/) (2026)
- [AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research](https://mlsanity.com/paper/2507.08038/) (2025)

## Abstract (arXiv)

Thousands of new scientific papers are published each month. Such information overload complicates researcher efforts to stay current with the state-of-the-art as well as to verify and correctly attribute claims. We pose the following research question: Given a text excerpt referencing a paper, could an LM act as a research assistant to correctly identify the referenced paper? We advance efforts to answer this question by building a benchmark that evaluates the abilities of LMs in citation attribution. Our benchmark, CiteME, consists of text excerpts from recent machine learning papers, each referencing a single other paper. CiteME use reveals a large gap between frontier LMs and human performance, with LMs achieving only 4.2-18.5% accuracy and humans 69.7%. We close this gap by introducing CiteAgent, an autonomous system built on the GPT-4o LM that can also search and read papers, which achieves an accuracy of 35.3% on CiteME. Overall, CiteME serves as a challenging testbed for open-ended claim attribution, driving the research community towards a future where any claim made by an LM can be automatically verified and discarded if found to be incorrect.