# Reflective Paper-to-Code Reproduction Enabled by Fine-Grained Verification

arXiv: https://arxiv.org/abs/2508.16671 · HTML: https://mlsanity.com/paper/2508.16671/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 21 Aug 2025.

## Builds on (most cited first, 15 of 15)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [AIDE: AI-Driven Exploration in the Space of Code](https://mlsanity.com/paper/2502.13138/) (2025)
- [AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls](https://mlsanity.com/paper/2402.04253/) (2024)
- [CodeRAG-Bench: Can Retrieval Augment Code Generation?](https://mlsanity.com/paper/2406.14497/) (2024)
- [Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning](https://mlsanity.com/paper/2504.17192/) (2025)
- [Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers](https://mlsanity.com/paper/2406.14325/) (2024)
- [Reproducibility in Machine Learning-Driven Research](https://mlsanity.com/paper/2307.10320/) (2023)
- [ResearchCodeAgent: An LLM Multi-Agent System for Automated Codification of Research Methodologies](https://mlsanity.com/paper/2504.20117/) (2025)
- [RAG or Fine-tuning? A Comparative Study on LCMs-based Code Completion in Industry](https://mlsanity.com/paper/2505.15179/) (2025)

## Built on by (most cited first, 10 of 10)

- [Toward Autonomous Long-Horizon Engineering for ML Research](https://mlsanity.com/paper/2604.13018/) (2026)
- [NERFIFY: A Multi-Agent Framework for Turning NeRF Papers into Code](https://mlsanity.com/paper/2603.00805/) (2026)
- [ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues](https://mlsanity.com/paper/2606.18237/) (2026)
- [Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills](https://mlsanity.com/paper/2609.02749/) (2026)
- [What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representations](https://mlsanity.com/paper/2510.17795/) (2025)
- [Enhancing Automated Paper Reproduction via Prompt-Free Collaborative Agents](https://mlsanity.com/paper/2512.02812/) (2025)
- [SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction](https://mlsanity.com/paper/2608.24252/) (2026)
- [ReproAgent: Contract-Guided Paper-to-Code Reproduction](https://mlsanity.com/paper/2608.24291/) (2026)
- [RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?](https://mlsanity.com/paper/2609.28850/) (2026)
- [SPLATIFY: Reproduce, Discover, Innovate! From Papers and Ideas to Trainable 3DGS Code](https://mlsanity.com/paper/2610.09116/) (2026)

## Abstract (arXiv)

Reproducing machine learning papers is essential for scientific progress but remains challenging for both humans and automated agents. Existing agent-based methods often struggle to fully and accurately reproduce implementation details such as mathematical formulas and algorithmic logic. Previous studies show that reflection with explicit feedback improves agent performance. However, current paper reproduction methods fail to effectively adopt this strategy. This gap mainly arises from the diverse paper patterns, complex method modules, and varied configurations encountered in research papers. Motivated by how humans use systematic checklists to efficiently debug complex code, we propose RePro, a Reflective Paper-to-Code Reproduction framework that automatically extracts a paper's fingerprint, referring to a comprehensive set of accurate and atomic criteria serving as high-quality supervisory signals. The framework first generates code based on the extracted information, and then leverages the fingerprint within iterative verification and refinement loop. This approach systematically detects discrepancies and produces targeted revisions to align generated code with the paper's implementation details. Extensive experiments on the PaperBench Code-Dev benchmark have been conducted, RePro achieves 13.0% performance gap over baselines, and it correctly revises complex logical and mathematical criteria in reflecting, on which the effectiveness is obvious.