mlsanityData to 7 Oct 2026

RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedback

arXiv 2510.06186 ↗First posted 7 Oct 2025cs.CL cs.AIcited by 24

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Large language models (LLMs) show the promise in supporting scientific research implementation, yet their ability to generate correct and executable code remains limited. Existing works largely adopt one-shot settings, ignoring the iterative and feedback-driven nature of realistic workflows of scientific research development. To address this gap, we present RECODE-H, a benchmark of 102 tasks from research papers and repositories that evaluates LLM agents through multi-turn interactions with LLM-simulated human feedback. It includes structured instructions,unit tests, and a five-level feedback hierarchy to reflect realistic researcher-agent collaboration. We further present ReCodeAgent, a framework that integrates feedback into iterative code generation. Experiments with leading LLMs, including GPT-5, Claude-Sonnet-4, DeepSeek-V3.1, and Gemini 2.5, show substantial performance gains with richer feedback, while also highlighting ongoing challenges in the generation of complex research code. RECODE-H establishes a foundation for developing adaptive, feedback-driven LLM agents in scientific research implementation

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 59 references that are in the map. First posted on arXiv 7 Oct 2025.

Builds on

38 of 59 references.

References · 38

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2023Teaching Large Language Models to Self-Debugcited by 963
  7. 2021Measuring Coding Challenge Competence With APPScited by 879
  8. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  9. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  10. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  11. 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 348
  12. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
26 more
  1. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  2. 2025PaperBench: Evaluating AI's Ability to Replicate AI Researchcited by 286
  3. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  4. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 208
  5. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  6. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  7. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  8. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 116
  9. 2023SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Researchcited by 112
  10. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  11. 2024EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  12. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  13. 2023A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trendscited by 66
  14. 2025AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-benchcited by 65
  15. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 42
  16. 2025EXP-Bench: Can AI Conduct AI Research Experiments?cited by 36
  17. 2025ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Codecited by 34
  18. 2024Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototypingcited by 29
  19. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Researchcited by 29
  20. 2025AI Scientists Fail Without Strong Implementation Capabilitycited by 20
  21. 2025RExBench: Can coding agents autonomously implement AI research extensions?cited by 17
  22. 2025ML-Dev-Bench: Comparative Analysis of AI Agents on ML development workflowscited by 16
  23. 2025CoRT: Code-integrated Reasoning within Thinkingcited by 15
  24. 2025ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environmentscited by 13
  25. 2025Code Benchmarks Should Prioritize Rigor, Reliability, and Reproducibilitycited by 10
  26. 2025Advancing the Scientific Method with Large Language Models: From Hypothesis to Discoverycited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 22.

24 papers.

Sort:

By area:

  1. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  2. 2026SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Taskscited by 24
  3. 2026ResearchGym: Evaluating Language Model Agents on Real-World AI Researchcited by 15
  4. 2026SWE-Together: Evaluating Coding Agents in Interactive User Sessionscited by 14
  5. 2026When Users Change Their Mind: Evaluating Interruptible Agents in Long-Horizon Web Navigationcited by 12
  6. 2025AdmTree: Compressing Lengthy Context with Adaptive Semantic Treescited by 7
  7. 2026EvoConfig: Self-Evolving Multi-Agent Systems for Efficient Autonomous Environment Configurationcited by 7
  8. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  9. 2025Embracing Trustworthy Brain-Agent Collaboration as Paradigm Extension for Intelligent Assistive Technologiescited by 5
  10. 2026When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agentscited by 5
  11. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system securitycited by 5
  12. 2026RSIAgent: Autonomous Exploration for Recursive Self-improvement in New Environmentscited by 5
12 more
  1. 2026Bridging the Agent-World Gap: Text World Models for LLM-based Agentscited by 3
  2. 2026When Only the Final Text Survives: Implicit Execution Tracing for Multi-Agent Auditingcited by 2
  3. 2026Benchmarking PhD-Level Coding in 3D Geometric Computer Visioncited by 2
  4. 2026"Excuse me, may I say something..." CoLabScience, A Proactive AI Assistant for Biomedical Discovery and LLM-Expert Collaborationscited by 1
  5. 2026Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agentscited by 1
  6. 2026From Knowing to Acting: Benchmarking Self-Awareness Capability of LLM Agentscited by 0
  7. 2026HAS-Bench: Evaluating LLM-Based Human-Agent Systems under Configurable Human Participationcited by 0
  8. 2026From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilitiescited by 0
  9. 2026TimeEvo: Failure-Driven Self-Evolution of a Time Series Agentcited by 0
  10. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0
  11. 2026ReMCTS: Reflection-Enhanced Monte Carlo Tree Search for Code Generationcited by 0
  12. 2026SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interactioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026No Test Cases, No Problem: Distillation-Driven Code Generation for Scientific Workflows
  2. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solving
  3. 2025AInsteinBench: Benchmarking Coding Agents on Scientific Repositories
  4. 2026SciNav: A General Agent Framework for Scientific Coding Tasks
  5. 2025InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Research
  6. 2026HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution
  7. 2026Glite ARF: Verifier-Driven Research with Parallel LLM Coding Agents
  8. 2026CURATE: Leveraging LLM Agents to Compose, Catalog, and Deploy Reproducible Workflows