mlsanityData to 14 Oct 2026

SciCode: A Research Coding Benchmark Curated by Scientists

arXiv 2407.13168 ↗First posted 18 Jul 2024cs.AI cs.CLcited by 178

Not on the map. Why?

Cited by 76 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Since language models (LMs) now outperform average humans on many challenging tasks, it has become increasingly difficult to develop challenging, high-quality, and realistic evaluations. We address this issue by examining LMs' capabilities to generate code for solving real scientific research problems. Incorporating input from scientists and AI researchers in 16 diverse natural science sub-fields, including mathematics, physics, chemistry, biology, and materials science, we created a scientist-curated coding benchmark, SciCode. The problems in SciCode naturally factorize into multiple subproblems, each involving knowledge recall, reasoning, and code synthesis. In total, SciCode contains 338 subproblems decomposed from 80 challenging main problems. It offers optional descriptions specifying useful scientific background information and scientist-annotated gold-standard solutions and test cases for evaluation. Claude3.5-Sonnet, the best-performing model among those tested, can solve only 4.6% of the problems in the most realistic setting. We believe that SciCode demonstrates both contemporary LMs' progress towards becoming helpful scientific assistants and sheds light on the development and evaluation of scientific AI in the future.

Key facts. Cited by 178 papers in the map (data through 14 Oct 2026). Builds on 27 of its 85 references that are in the map. First posted on arXiv 18 Jul 2024.

Builds on

27 of 85 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2021Program Synthesis with Large Language Modelscited by 3,637
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  10. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  11. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  12. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
15 more
  1. 2021Measuring Coding Challenge Competence With APPScited by 902
  2. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  3. 2023StarCoder: may the source be with you!cited by 799
  4. 2022PAL: Program-aided Language Modelscited by 652
  5. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  6. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  7. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 452
  8. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  9. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
  10. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  11. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 183
  12. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  13. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 158
  14. 2023The Impact of Large Language Models on Scientific Discovery: a Preliminary Study using GPT-4cited by 112
  15. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 87

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 61; 2026: 109.

178 papers.

Sort:

By area:

  1. 2026Kimi K2.5: Visual Agentic Intelligencecited by 501
  2. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 467
  3. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 363
  4. 2026Gemma 4 Technical Reportcited by 259
  5. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 235
  6. 2025Survey on Evaluation of LLM-based Agentscited by 204
  7. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  8. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 166
  9. 2025Towards a Science of Scaling Agent Systemscited by 138
  10. 2026Kimi K3: Open Frontier Intelligencecited by 138
  11. 2024CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmarkcited by 123
  12. 2025SWE-Lancer: Can Frontier LLMs Earn 1 Million from Real-World Freelance Software Engineering?cited by 105
38 more
  1. 2025Towards Scientific Intelligence: A Survey of LLM-based Scientific Agentscited by 98
  2. 2024Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agentscited by 97
  3. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 92
  4. 2025From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discoverycited by 77
  5. 2025How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inferencecited by 76
  6. 2025SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agentscited by 71
  7. 2025MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Researchcited by 64
  8. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 52
  9. 2025Automated Hypothesis Validation with Agentic Sequential Falsificationscited by 44
  10. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 42
  11. 2025CodePDE: An Inference Framework for LLM-driven PDE Solver Generationcited by 39
  12. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 38
  13. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 36
  14. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 32
  15. 2026Towards Execution-Grounded Automated AI Researchcited by 28
  16. 2025AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?cited by 27
  17. 2025FEABench: Evaluating Language Models on Multiphysics Reasoning Abilitycited by 22
  18. 2026BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?cited by 22
  19. 2026Scaling the Horizon, Not the Parameters: Reaching Trillion-Parameter Performance with a 35B Agentcited by 22
  20. 2025REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?cited by 21
  21. 2025Probing the Critical Point (CritPt) of AI Reasoning: a Frontier Physics Research Benchmarkcited by 21
  22. 2024Literature Meets Data: A Synergistic Approach to Hypothesis Generationcited by 20
  23. 2025All That Glitters is Not Novel: Plagiarism in AI Generated Researchcited by 18
  24. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 18
  25. 2025YESciEval: Robust LLM-as-a-Judge for Scientific Question Answeringcited by 16
  26. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 16
  27. 2026Effective Strategies for Asynchronous Software Engineering Agentscited by 16
  28. 2025ResearchCodeAgent: An LLM Multi-Agent System for Automated Codification of Research Methodologiescited by 15
  29. 2025PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priorscited by 15
  30. 2025How Far Are AI Scientists from Changing the World?cited by 15
  31. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  32. 2025AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvementcited by 14
  33. 2025AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesiscited by 14
  34. 2025AstroVisBench: A Code Benchmark for Scientific Computing and Visualization in Astronomycited by 13
  35. 2026DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Usecited by 13
  36. 2026HeurekaBench: A Benchmarking Framework for AI Co-scientistcited by 12
  37. 2024From Generalist to Specialist: A Survey of Large Language Models for Chemistrycited by 11
  38. 2025AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientistscited by 11

50 most cited of 178 All 178 (JSON)