# AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Academic Papers

arXiv: https://arxiv.org/abs/2504.20115 · HTML: https://mlsanity.com/paper/2504.20115/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 31 of its 62 references that are in the map.
- First posted on arXiv 28 Apr 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 31)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges](https://mlsanity.com/paper/2401.07339/) (2024)
- [MinerU: An Open-Source Solution for Precise Document Content Extraction](https://mlsanity.com/paper/2409.18839/) (2024)
- [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023)
- [Nougat: Neural Optical Understanding for Academic Documents](https://mlsanity.com/paper/2308.13418/) (2023)
- [AutoAgents: A Framework for Automatic Agent Generation](https://mlsanity.com/paper/2309.17288/) (2023)
- [General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model](https://mlsanity.com/paper/2409.01704/) (2024)
- [Auto-Keras: An Efficient Neural Architecture Search System](https://mlsanity.com/paper/1806.10282/) (2018)
- [VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks](https://mlsanity.com/paper/2406.08394/) (2024)
- [Evaluation of OpenAI o1: Opportunities and Challenges of AGI](https://mlsanity.com/paper/2409.18486/) (2024)
- [SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers](https://mlsanity.com/paper/2407.09413/) (2024)
- [Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning](https://mlsanity.com/paper/2504.17192/) (2025)
- [MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems](https://mlsanity.com/paper/2404.09486/) (2024)
- [CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generation](https://mlsanity.com/paper/2501.07811/) (2025)
- [Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks](https://mlsanity.com/paper/2403.04814/) (2024)
- [Ocean-OCR: Towards General OCR Application via a Vision-Language Model](https://mlsanity.com/paper/2501.15558/) (2025)
- [Amazon SageMaker Autopilot: a white box AutoML solution at scale](https://mlsanity.com/paper/2012.08483/) (2020)
- [From Code to Courtroom: LLMs as the New Software Judges](https://mlsanity.com/paper/2503.02246/) (2025)
- [cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers](https://mlsanity.com/paper/2406.08398/) (2024)
- [RES-Q: Evaluating Code-Editing Large Language Model Systems at the Repository Scale](https://mlsanity.com/paper/2406.16801/) (2024)
- [L^2GC:Lorentzian Linear Graph Convolutional Networks for Node Classification](https://mlsanity.com/paper/2403.06064/) (2024)
- [CodeRefine: A Pipeline for Enhancing LLM-Generated Code Implementations of Research Papers](https://mlsanity.com/paper/2408.13366/) (2024)
- [EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decomposition](https://mlsanity.com/paper/2501.12067/) (2025)

## Built on by (most cited first, 20 of 20)

- [RPG: A Repository Planning Graph for Unified and Scalable Codebase Generation](https://mlsanity.com/paper/2509.16198/) (2025)
- [Scientific Algorithm Discovery by Augmenting AlphaEvolve with Deep Research](https://mlsanity.com/paper/2510.06056/) (2025)
- [The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements](https://mlsanity.com/paper/2506.22419/) (2025)
- [How Far Are AI Scientists from Changing the World?](https://mlsanity.com/paper/2507.23276/) (2025)
- [Sculpting the Vector Space: Towards Efficient Multi-Vector Visual Document Retrieval via Prune-then-Merge Framework](https://mlsanity.com/paper/2602.19549/) (2026)
- [NanoResearch: Co-Evolving Skills, Memory, and Policy for Personalized Research Automation](https://mlsanity.com/paper/2605.10813/) (2026)
- [NERFIFY: A Multi-Agent Framework for Turning NeRF Papers into Code](https://mlsanity.com/paper/2603.00805/) (2026)
- [SciPredict: Can LLMs Predict the Outcomes of Scientific Experiments in Natural Sciences?](https://mlsanity.com/paper/2604.10718/) (2026)
- [From paper to benchmark: agentic, framework-based reproduction of under-specified methods in machine health intelligence](https://mlsanity.com/paper/2605.28371/) (2026)
- [Evaluating the Effectiveness of Large Language Models in Solving Simple Programming Tasks: A User-Centered Study](https://mlsanity.com/paper/2507.04043/) (2025)
- [Enhancing Automated Paper Reproduction via Prompt-Free Collaborative Agents](https://mlsanity.com/paper/2512.02812/) (2025)
- [REVERE: Reflective Evolving Research Engineer](https://mlsanity.com/paper/2603.20667/) (2026)
- [SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction](https://mlsanity.com/paper/2608.24252/) (2026)
- [ReproAgent: Contract-Guided Paper-to-Code Reproduction](https://mlsanity.com/paper/2608.24291/) (2026)
- [Lifecycle-Aware code generation: Leveraging Software Engineering Phases in LLMs](https://mlsanity.com/paper/2510.24019/) (2025)
- [SoK: Understanding (New) Security Issues Across AI4Code Use Cases](https://mlsanity.com/paper/2512.18456/) (2025)
- [SW-A2-Bench: Benchmarking Autonomous Software Agent Generation for Agentic Web](https://mlsanity.com/paper/2604.04226/) (2026)
- [RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?](https://mlsanity.com/paper/2609.28850/) (2026)
- [AECP: Artifact-Exclusive Communication Protocol for Multi-Agent Code Generation](https://mlsanity.com/paper/2610.06481/) (2026)
- [SPLATIFY: Reproduce, Discover, Innovate! From Papers and Ideas to Trainable 3DGS Code](https://mlsanity.com/paper/2610.09116/) (2026)

## Abstract (arXiv)

Machine Learning (ML) research is spread through academic papers featuring rich multimodal content, including text, diagrams, and tabular results. However, translating these multimodal elements into executable code remains a challenging and time-consuming process that requires substantial ML expertise. We introduce ``Paper-to-Code'' (P2C), a novel task that transforms the multimodal content of scientific publications into fully executable code repositories, which extends beyond the existing formulation of code generation that merely converts textual descriptions into isolated code snippets. To automate the P2C process, we propose AutoP2C, a multi-agent framework based on large language models that processes both textual and visual content from research papers to generate complete code repositories. Specifically, AutoP2C contains four stages: (1) repository blueprint extraction from established codebases, (2) multimodal content parsing that integrates information from text, equations, and figures, (3) hierarchical task decomposition for structured code generation, and (4) iterative feedback-driven debugging to ensure functionality and performance. Evaluation on a benchmark of eight research papers demonstrates the effectiveness of AutoP2C, which can successfully generate executable code repositories for all eight papers, while OpenAI-o1 or DeepSeek-R1 can only produce runnable code for one paper. The code is available at https://github.com/shoushouyu/Automated-Paper-to-Code.