# From paper to benchmark: agentic, framework-based reproduction of under-specified methods in machine health intelligence

arXiv: https://arxiv.org/abs/2605.28371 · HTML: https://mlsanity.com/paper/2605.28371/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 27 May 2026.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)

## Builds on (most cited first, 23 of 23)

- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [Agent Laboratory: Using LLM Agents as Research Assistants](https://mlsanity.com/paper/2501.04227/) (2025)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [AIDE: AI-Driven Exploration in the Space of Code](https://mlsanity.com/paper/2502.13138/) (2025)
- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning](https://mlsanity.com/paper/2504.17192/) (2025)
- [SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Papers](https://mlsanity.com/paper/2504.00255/) (2025)
- [ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code](https://mlsanity.com/paper/2506.02314/) (2025)
- [AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage](https://mlsanity.com/paper/2505.20662/) (2025)
- [Machine learning pipeline for battery state of health estimation](https://mlsanity.com/paper/2102.00837/) (2021)
- [TS-Reasoner: Domain-Oriented Time Series Inference Agents for Reasoning and Automated Analysis](https://mlsanity.com/paper/2410.04047/) (2024)
- [TimeART: Towards Agentic Time Series Reasoning via Tool-Augmentation](https://mlsanity.com/paper/2601.13653/) (2026)
- [AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Academic Papers](https://mlsanity.com/paper/2504.20115/) (2025)
- [Argos: Agentic Time-Series Anomaly Detection with Autonomous Rule Generation via Large Language Models](https://mlsanity.com/paper/2501.14170/) (2025)
- [From Reproduction to Replication: Evaluating Research Agents with Progressive Code Masking](https://mlsanity.com/paper/2506.19724/) (2025)
- [TS-Agent: Understanding and Reasoning Over Raw Time Series via Iterative Insight Gathering](https://mlsanity.com/paper/2510.07432/) (2025)
- [A Comparison of Residual-based Methods on Fault Detection](https://mlsanity.com/paper/2309.02274/) (2023)
- [Fault Prognosis of Turbofan Engines: Eventual Failure Prediction and Remaining Useful Life Estimation](https://mlsanity.com/paper/2303.12982/) (2023)
- [CodeRefine: A Pipeline for Enhancing LLM-Generated Code Implementations of Research Papers](https://mlsanity.com/paper/2408.13366/) (2024)
- [AutoIAD: Manager-Driven Multi-Agent Collaboration for Automated Industrial Anomaly Detection](https://mlsanity.com/paper/2508.05503/) (2025)
- [Time Series Reasoning via Process-Verifiable Thinking Data Synthesis and Scheduling for Tailored LLM Reasoning](https://mlsanity.com/paper/2602.07830/) (2026)
- [Do Not Treat Code as Natural Language: Implications for Repository-Level Code Generation and Beyond](https://mlsanity.com/paper/2602.11671/) (2026)
- [To Charge or to Sell? EV Pack Useful Life Estimation via LSTMs, CNNs, and Autoencoders](https://mlsanity.com/paper/2110.03585/) (2021)

## Built on by (most cited first, 2 of 2)

- [Towards Unified and Data-Efficient Prognostics and Health Management with Tabular Foundation Models](https://mlsanity.com/paper/2606.05481/) (2026)
- [RSI-Forge: From Research Papers to Environments for Recursive Self-Improvement](https://mlsanity.com/paper/2610.09426/) (2026)

## Abstract (arXiv)

Industrial Prognostics and Health Management (PHM) provides a representative case study for a broader challenge in applied machine learning: translating published papers into executable, benchmark-ready implementations. Reproducing under-specified methods in PHM is particularly difficult due to restricted access to industrial datasets, incomplete reporting of preprocessing and evaluation protocols, and implicit design choices (e.g., windowing, target construction, data splits) that critically affect performance. Existing paper-to-code systems generate implementations for individual papers, but these artifacts are often not directly comparable due to inconsistencies in assumptions and evaluation settings. We introduce agentic, framework-based PHM paper reproduction, where an agent translates a paper into a shared PHM benchmark framework via a slot-binding interface. This interface maps equations and protocol descriptions into structured components (task definitions, dataset adapters, windowing, targets, models, and evaluators), while explicitly recording unresolved assumptions. The resulting implementations are validated against standardized task contracts and evaluation hooks, enabling consistent and comparable benchmarking. We evaluate this approach on 16 PHM papers, comparing framework-enhanced, skill-based and prompt-based agentic reproduction against a recent framework-free paper-reproduction agent. We assess reproduction success, model-based code evaluation, framework binding of paper assumptions, and cross-paper benchmark comparability under standardized protocols. Our results show that coupling agentic generation with a shared framework transforms paper reproduction from isolated code synthesis into executable, assumption-aware, and systematically comparable benchmark implementations.