mlsanityData to 14 Oct 2026

CiteME: Can Language Models Accurately Cite Scientific Claims?

arXiv 2407.12861 ↗First posted 10 Jul 2024cs.CL cs.AI cs.HCcited by 48

Not on the map. Why?

Abstract (arXiv)

Thousands of new scientific papers are published each month. Such information overload complicates researcher efforts to stay current with the state-of-the-art as well as to verify and correctly attribute claims. We pose the following research question: Given a text excerpt referencing a paper, could an LM act as a research assistant to correctly identify the referenced paper? We advance efforts to answer this question by building a benchmark that evaluates the abilities of LMs in citation attribution. Our benchmark, CiteME, consists of text excerpts from recent machine learning papers, each referencing a single other paper. CiteME use reveals a large gap between frontier LMs and human performance, with LMs achieving only 4.2-18.5% accuracy and humans 69.7%. We close this gap by introducing CiteAgent, an autonomous system built on the GPT-4o LM that can also search and read papers, which achieves an accuracy of 35.3% on CiteME. Overall, CiteME serves as a challenging testbed for open-ended claim attribution, driving the research community towards a future where any claim made by an LM can be automatically verified and discarded if found to be incorrect.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 60 of its 107 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

60 of 107 references.

References · 60

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
38 more
  1. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  2. 2019Language Models as Knowledge Bases?cited by 1,936
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  4. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  5. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  6. 2019Green AIcited by 627
  7. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  8. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 430
  9. 2023Augmented Language Models: a Surveycited by 403
  10. 2023How Language Model Hallucinations Can Snowballcited by 311
  11. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  12. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225
  13. 2015A Neural Conversational Modelcited by 196
  14. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
  15. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 149
  16. 2018A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applicationscited by 136
  17. 2020Generating Fact Checking Explanationscited by 132
  18. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 129
  19. 2023Factuality Challenges in the Era of Large Language Modelscited by 126
  20. 2023The Semantic Scholar Open Data Platformcited by 118
  21. 2023Complex Claim Verification with Evidence Retrieved in the Wildcited by 110
  22. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 97
  23. 2022SciFact-Open: Towards open-domain scientific claim verificationcited by 94
  24. 2013Deep Learning of Representations: Looking Forwardcited by 84
  25. 2019A Richly Annotated Corpus for Different Tasks in Automated Fact-Checkingcited by 74
  26. 2019Target-Guided Open-Domain Conversationcited by 66
  27. 2022Generating Scientific Claims for Zero-Shot Scientific Fact Checkingcited by 59
  28. 2020Convergence of Sparse Variational Inference in Gaussian Processes Regressioncited by 53
  29. 2015Traversing Knowledge Graphs in Vector Spacecited by 51
  30. 2023Effective Large Language Model Adaptation for Improved Grounding and Citation Generationcited by 45
  31. 2018Content-Based Citation Recommendationcited by 43
  32. 2024Training Language Models to Generate Text with Citations via Fine-grained Rewardscited by 42
  33. 2024Are LLMs Ready for Real-World Materials Discovery?cited by 40
  34. 2016Deep Gaussian Processes for Regression using Approximate Expectation Propagationcited by 39
  35. 2020Citation Recommendation: Approaches and Datasetscited by 36
  36. 2024Source-Aware Training Enables Knowledge Attribution in Language Modelscited by 30
  37. 2019A Context-Aware Citation Recommendation Model with BERT and Graph Convolutional Networkscited by 29
  38. 2023CiteSee: Augmenting Citations in Scientific Papers with Persistent and Personalized Historical Contextcited by 22

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 20; 2026: 23.

48 papers.

Sort:

By area:

  1. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  2. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  3. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  4. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 123
  5. 2025Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directionscited by 77
  6. 2025AgentRxiv: Towards Collaborative Autonomous Researchcited by 72
  7. 2025Automated Hypothesis Validation with Agentic Sequential Falsificationscited by 44
  8. 2025HypoBench: Towards Systematic and Principled Benchmarking for Hypothesis Generationcited by 37
  9. 2024Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refusecited by 34
  10. 2024Literature Meets Data: A Synergistic Approach to Hypothesis Generationcited by 20
  11. 2025IRIS: Interactive Research Ideation System for Accelerating Scientific Discoverycited by 19
  12. 2026ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidencecited by 19
36 more
  1. 2025How Far Are AI Scientists from Changing the World?cited by 15
  2. 2026Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agentscited by 12
  3. 2026AI for Auto-Research: Roadmap & User Guidecited by 11
  4. 2025How Deep Do Large Language Models Internalize Scientific Literature and Citation Practices?cited by 10
  5. 2026AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discoverycited by 10
  6. 2025HoT: Highlighted Chain of Thought for Referencing Supporting Facts from Inputscited by 9
  7. 2025LLM-REVal: Can We Trust LLM Reviewers Yet?cited by 9
  8. 2026BibAgent: An Agentic Framework for Traceable Miscitation Detection in Scientific Literaturecited by 9
  9. 2025VeriCite: Towards Reliable Citations in Retrieval-Augmented Generation via Rigorous Verificationcited by 8
  10. 2026LABBench2: An Improved Benchmark for AI Systems Performing Biology Researchcited by 8
  11. 2025Can Language Models Falsify? Evaluating Algorithmic Reasoning with Counterexample Creationcited by 7
  12. 2025XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaborationcited by 7
  13. 2025FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modescited by 7
  14. 2026Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agentscited by 6
  15. 2025Large Language Model Sourcing: A Surveycited by 5
  16. 2026ScholarGym: Benchmarking Large Language Model Capabilities in the Information-Gathering Stage of Deep Researchcited by 3
  17. 2026A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversationscited by 3
  18. 2025AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Researchcited by 2
  19. 2025Copy-Paste to Mitigate Large Language Model Hallucinationscited by 2
  20. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  21. 2026IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answeringcited by 2
  22. 2026Accelerating Scientific Research with Gemini in the Real-Worldcited by 2
  23. 2025Bonsai: Interpretable Tree-Adaptive Grounded Reasoningcited by 1
  24. 2026Power and Limitations of Aggregation in Compound AI Systemscited by 1
  25. 2026RUBEN: Rule-Based Explanations for Retrieval-Augmented LLM Systemscited by 1
  26. 2026CiteCheck: Retrieval-Grounded Detection of LLM Citation Hallucinations in Scientific Textcited by 1
  27. 2026MindFlow: Mind Supernet Powered Thinking Flows for Research Idea Innovationcited by 1
  28. 2025Rule-Based Explanations for Retrieval-Augmented LLM Systemscited by 0
  29. 2026Detecting Miscitation on the Scholarly Web through LLM-Augmented Text-Rich Graph Learningcited by 0
  30. 2026Attribution Bias in Large Language Modelscited by 0
  31. 2026Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attributioncited by 0
  32. 2026Towards a new paradigm of scientific discovery with socialized artificial intelligencecited by 0
  33. 2026Is this Citation on Point?cited by 0
  34. 2026Citing Less Critically: LLMs Reshape the Rhetoric and Reach of Scientific Citationcited by 0
  35. 2026ATTRICITE: Training an Open 4B Model for Citation Recovery toward Faithful Attributioncited by 0
  36. 2026SciCurateBench: Curating Experimental Data from Heterogeneous Scientific Literature with Verbal-Action Agentscited by 0