# What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representations

arXiv: https://arxiv.org/abs/2510.17795 · HTML: https://mlsanity.com/paper/2510.17795/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 20 Oct 2025.

## Builds on (most cited first, 16 of 16)

- [MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering](https://mlsanity.com/paper/2410.07095/) (2024)
- [A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity](https://mlsanity.com/paper/2401.01967/) (2024)
- [Deep Research Agents: A Systematic Examination And Roadmap](https://mlsanity.com/paper/2506.18096/) (2025)
- [Stay on topic with Classifier-Free Guidance](https://mlsanity.com/paper/2306.17806/) (2023)
- [Test-Time Model Adaptation with Only Forward Passes](https://mlsanity.com/paper/2404.01650/) (2024)
- [AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench](https://mlsanity.com/paper/2507.02554/) (2025)
- [AI4Research: A Survey of Artificial Intelligence for Scientific Research](https://mlsanity.com/paper/2507.01903/) (2025)
- [EXP-Bench: Can AI Conduct AI Research Experiments?](https://mlsanity.com/paper/2505.24785/) (2025)
- [AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage](https://mlsanity.com/paper/2505.20662/) (2025)
- [Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodings](https://mlsanity.com/paper/2402.17135/) (2024)
- [LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research](https://mlsanity.com/paper/2506.17335/) (2025)
- [Paper2Agent: Reimagining Research Papers As Interactive and Reliable AI Agents](https://mlsanity.com/paper/2509.06917/) (2025)
- [AutoMind: Adaptive Knowledgeable Agent for Automated Data Science](https://mlsanity.com/paper/2506.10974/) (2025)
- [RExBench: Can coding agents autonomously implement AI research extensions?](https://mlsanity.com/paper/2506.22598/) (2025)
- [Reflective Paper-to-Code Reproduction Enabled by Fine-Grained Verification](https://mlsanity.com/paper/2508.16671/) (2025)
- [Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization](https://mlsanity.com/paper/2509.09321/) (2025)

## Built on by (most cited first, 1 of 1)

- [The Last Human-Written Paper: Agent-Native Research Artifacts](https://mlsanity.com/paper/2604.24658/) (2026)

## Abstract (arXiv)

Replicating AI research is a crucial yet challenging task for large language model (LLM) agents. Existing approaches often struggle to generate executable code, primarily due to insufficient background knowledge and the limitations of retrieval-augmented generation (RAG) methods, which fail to capture latent technical details hidden in referenced papers. Furthermore, previous approaches tend to overlook valuable implementation-level code signals and lack structured knowledge representations that support multi-granular retrieval and reuse. To overcome these challenges, we propose Executable Knowledge Graphs (xKG), a pluggable, paper-centric knowledge base that automatically integrates code snippets and technical insights extracted from scientific literature. When integrated into three agent frameworks with two different LLMs, xKG shows substantial performance gains (10.9% with o3-mini) on PaperBench, demonstrating its effectiveness as a general and extensible solution for automated AI research replication. Code is available at https://github.com/zjunlp/xKG.