mlsanityData to 14 Oct 2026

SciCoQA: Quality Assurance for Scientific Paper--Code Alignment

arXiv 2601.12910 ↗First posted 19 Jan 2026cs.CL cs.AIcited by 10

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Discrepancies between scientific papers and their code undermine reproducibility, a concern that grows as automated research agents scale scientific output beyond human review capacity. Whether LLMs can reliably detect such discrepancies has not been systematically measured. To this end, we present SciCoQA, a dataset of 635 paper-code discrepancies (92 real, 543 synthetic) for this cross-modal verification task. Across 22 evaluated models, even the best-performing LLMs, Gemini 3.1 Pro and GPT-5 Mini, detect only 46.7% of real-world discrepancies, revealing a critical gap in automated scientific quality assurance. We construct SciCoQA from GitHub issues and reproducibility papers, and propose a synthetic generation pipeline to scale beyond AI to Physics, Quantitative Biology, and other computational sciences. We further introduce a taxonomy of discrepancy types and categories to characterize the occurring mismatches. Our analysis shows that models particularly struggle with omitted paper details, long-context inputs, and papers outside their pre-training corpus.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 49 of its 95 references that are in the map. First posted on arXiv 19 Jan 2026.

Builds on

49 of 95 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025ResearchCodeAgent: An LLM Multi-Agent System for Automated Codification of Research Methodologies28 Apr
  2. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers31 Mar
  3. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark17 Sep
  4. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers6 Sep

Other references · 45

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  9. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  10. 2017Deep Reinforcement Learning that Matterscited by 754
  11. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  12. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
33 more
  1. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  2. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  3. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
  4. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  5. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 322
  6. 2024ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Modelscited by 210
  7. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 169
  8. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  9. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 139
  10. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 122
  11. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 119
  12. 2018Troubling Trends in Machine Learning Scholarshipcited by 108
  13. 2025From Automation to Autonomy: A Survey on Large Language Models in Scientific Discoverycited by 93
  14. 2019A Step Toward Quantifying Independently Reproducible Machine Learning Researchcited by 77
  15. 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 71
  16. 2025Paper2Code: Automating Code Generation from Scientific Papers in Machine Learningcited by 70
  17. 2025DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressivelycited by 63
  18. 2024Large Language Models Can Self-Correct with Key Condition Verificationcited by 39
  19. 2024RepoQA: Evaluating Long Context Code Understandingcited by 35
  20. 2020What Can We Do to Improve Peer Review in NLP?cited by 32
  21. 2025Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?cited by 32
  22. 2025When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Researchcited by 26
  23. 2025EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checkingcited by 24
  24. 2025Devstral: Fine-tuning Language Models for Coding Agent Applicationscited by 24
  25. 2020Deep Just-In-Time Inconsistency Detection Between Comments and Source Codecited by 20
  26. 2025FLAWS: A Benchmark for Error Identification and Localization in Scientific Paperscited by 14
  27. 2025Reviewing Scientific Papers for Critical Problems With Reasoning LLMs: Baseline Approaches and Automatic Evaluationcited by 12
  28. 2025Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Frameworkcited by 11
  29. 2017Sustainable computational science: the ReScience initiativecited by 10
  30. 2024WithdrarXiv: A Large-Scale Dataset for Retraction Studycited by 9
  31. 2022Code Comment Inconsistency Detection with BERT and Longformercited by 6
  32. 2025To Err Is Human: Systematic Quantification of Errors in Published AI Papers via LLM Analysiscited by 5
  33. 2023DocChecker: Bootstrapping Code Large Language Model for Detecting and Resolving Code-Comment Inconsistenciescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026The Last Human-Written Paper: Agent-Native Research Artifactscited by 16
  2. 2026When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agentscited by 6
  3. 2026Preventing the Collapse of Peer Review Requires Verification-First AIcited by 3
  4. 2026Do Papers Tell the Whole Story? A Benchmark and Framework for Uncovering Hidden Implementation Gaps in Bioinformaticscited by 1
  5. 2026SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproductioncited by 1
  6. 2026ReproAgent: Contract-Guided Paper-to-Code Reproductioncited by 1
  7. 2026Dude: A Dual-Detection Multi-Agent System for Paper-Code Discrepancy Detectioncited by 0
  8. 2026IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specificationscited by 0
  9. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0
  10. 2026YouRA: A Persistent-State Architecture for Evidence-Traceable Autonomous Research Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies
  2. 2026Coding-agents can replicate scientific machine learning papers
  3. 2026SciTrue: Reliable Scientific Claim Validation with Frontier and Open Language Models at the NTCIR SciClaimEval Task
  4. 2026VERITAS: Towards a General-Purpose Replication Tool for Scientific Research
  5. 2026Not Just Reason, Not Just Scan: Reinforcement Learning for Proactive Scientific Error Verification over Academic Paper
  6. 2025BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers?
  7. 2026Towards Automating Scientific Review with Google's Paper Assistant Tool
  8. 2026Deterministic Integrity Gates for LLM-Assisted Clinical Manuscript Preparation: An Auditable Biomedical Informatics Architecture