# ResearchCodeAgent: An LLM Multi-Agent System for Automated Codification of Research Methodologies arXiv: https://arxiv.org/abs/2504.20117 ยท HTML: https://mlsanity.com/paper/2504.20117/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. - Builds on 19 of its 71 references that are in the map. - First posted on arXiv 28 Apr 2025. ## Placed in - LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/) ## Builds on (most cited first, 19 of 19) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017) - [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024) - [MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation](https://mlsanity.com/paper/2310.03302/) (2023) - [ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery](https://mlsanity.com/paper/2410.05080/) (2024) - [ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models](https://mlsanity.com/paper/2404.07738/) (2024) - [SciMON: Scientific Inspiration Machines Optimized for Novelty](https://mlsanity.com/paper/2305.14259/) (2023) - [SciCode: A Research Coding Benchmark Curated by Scientists](https://mlsanity.com/paper/2407.13168/) (2024) - [Multi-lingual Evaluation of Code Generation Models](https://mlsanity.com/paper/2210.14868/) (2022) - [CodePlan: Repository-level Coding using LLMs and Planning](https://mlsanity.com/paper/2309.12499/) (2023) - [Large Language Models for Automated Open-domain Scientific Hypotheses Discovery](https://mlsanity.com/paper/2309.02726/) (2023) - [BLADE: Benchmarking Language Model Agents for Data-Driven Science](https://mlsanity.com/paper/2408.09667/) (2024) - [MLR-Copilot: Autonomous Machine Learning Research based on Large Language Models Agents](https://mlsanity.com/paper/2408.14033/) (2024) - [Robust Optimization as Data Augmentation for Large-scale Graphs](https://mlsanity.com/paper/2010.09891/) (2020) - [Acceleron: A Tool to Accelerate Research Ideation](https://mlsanity.com/paper/2403.04382/) (2024) - [You Only Need Half: Boosting Data Augmentation by Using Partial Content](https://mlsanity.com/paper/2405.02830/) (2024) ## Built on by (most cited first, 15 of 15) - [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025) - [AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage](https://mlsanity.com/paper/2505.20662/) (2025) - [Evaluating Memory Structure in LLM Agents](https://mlsanity.com/paper/2602.11243/) (2026) - [Reflective Paper-to-Code Reproduction Enabled by Fine-Grained Verification](https://mlsanity.com/paper/2508.16671/) (2025) - [SciCoQA: Quality Assurance for Scientific Paper--Code Alignment](https://mlsanity.com/paper/2601.12910/) (2026) - [Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap](https://mlsanity.com/paper/2608.05179/) (2026) - [A Human-AI Collaborative Workflow for Mathematical Discovery: A Case Study in Grover-Compatible Riemannian Optimization](https://mlsanity.com/paper/2512.09443/) (2025) - [NanoResearch: Co-Evolving Skills, Memory, and Policy for Personalized Research Automation](https://mlsanity.com/paper/2605.10813/) (2026) - [SAGE: Agentic Framework for Interpretable and Clinically Translatable Computational Pathology Biomarker Discovery](https://mlsanity.com/paper/2602.00953/) (2026) - [NERFIFY: A Multi-Agent Framework for Turning NeRF Papers into Code](https://mlsanity.com/paper/2603.00805/) (2026) - [An Agentic Approach Towards Replication Package Quality Evaluation](https://mlsanity.com/paper/2606.02006/) (2026) - [Coding-agents can replicate scientific machine learning papers](https://mlsanity.com/paper/2607.02134/) (2026) - [Evolving and Executing Research Plans via Double-Loop Multi-Agent Collaboration](https://mlsanity.com/paper/2510.06761/) (2025) - [From Execution to Capability: Scientific Experience Consolidation via Procedural Knowledge Synthesis](https://mlsanity.com/paper/2607.24459/) (2026) - [Training Numerical Intelligence via Auto-Diagnosis and Skill Discovery](https://mlsanity.com/paper/2610.03872/) (2026) ## Abstract (arXiv) In this paper we introduce ResearchCodeAgent, a novel multi-agent system leveraging large language models (LLMs) agents to automate the codification of research methodologies described in machine learning literature. The system bridges the gap between high-level research concepts and their practical implementation, allowing researchers auto-generating code of existing research papers for benchmarking or building on top-of existing methods specified in the literature with availability of partial or complete starter code. ResearchCodeAgent employs a flexible agent architecture with a comprehensive action suite, enabling context-aware interactions with the research environment. The system incorporates a dynamic planning mechanism, utilizing both short and long-term memory to adapt its approach iteratively. We evaluate ResearchCodeAgent on three distinct machine learning tasks with distinct task complexity and representing different parts of the ML pipeline: data augmentation, optimization, and data batching. Our results demonstrate the system's effectiveness and generalizability, with 46.9% of generated code being high-quality and error-free, and 25% showing performance improvements over baseline implementations. Empirical analysis shows an average reduction of 57.9% in coding time compared to manual implementation. We observe higher gains for more complex tasks. ResearchCodeAgent represents a significant step towards automating the research implementation process, potentially accelerating the pace of machine learning research.