# VERITAS: Towards a General-Purpose Replication Tool for Scientific Research

arXiv: https://arxiv.org/abs/2607.02931 · HTML: https://mlsanity.com/paper/2607.02931/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 3 Jul 2026.

## Builds on (most cited first, 15 of 15)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces](https://mlsanity.com/paper/2601.11868/) (2026)
- [PaperBench: Evaluating AI's Ability to Replicate AI Research](https://mlsanity.com/paper/2504.01848/) (2025)
- [CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark](https://mlsanity.com/paper/2409.11363/) (2024)
- [Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation](https://mlsanity.com/paper/2510.11977/) (2025)
- [Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning](https://mlsanity.com/paper/2504.17192/) (2025)
- [AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage](https://mlsanity.com/paper/2505.20662/) (2025)
- [REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?](https://mlsanity.com/paper/2507.18901/) (2025)
- [RExBench: Can coding agents autonomously implement AI research extensions?](https://mlsanity.com/paper/2506.22598/) (2025)
- [Compound Deception in Elite Peer Review: A Failure Mode Taxonomy of 100 Fabricated Citations at NeurIPS 2025](https://mlsanity.com/paper/2602.05930/) (2026)
- [AstroM3: A self-supervised multimodal model for astronomy](https://mlsanity.com/paper/2411.08842/) (2024)
- [ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences](https://mlsanity.com/paper/2602.11354/) (2026)
- [The Story is Not the Science: Execution-Grounded Evaluation of Mechanistic Interpretability Research](https://mlsanity.com/paper/2602.18458/) (2026)
- [Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results](https://mlsanity.com/paper/2604.21965/) (2026)
- [PaperRepro: Automated Computational Reproducibility Assessment for Social Science Papers](https://mlsanity.com/paper/2603.00058/) (2026)

## Built on by (most cited first, 2 of 2)

- [Training AI Scientists to Replicate Research](https://mlsanity.com/paper/2608.13331/) (2026)
- [ShadowMiner v1 - An Experience Report on Implementing and Measuring a Problem-and-Hypothesis Discovery Engine](https://mlsanity.com/paper/2610.04339/) (2026)

## Abstract (arXiv)

AI tools are accelerating scientific publication while the systems that review it struggle to keep up, and independent verification of published research has become both harder and more important. As manual replication is slow and expensive, a growing line of work uses coding agents to automate parts of the process. Existing efforts are largely packaged as benchmarks with companion agents that only run inside the benchmark's own pipeline, and no general-purpose replication tool exists. We present VERITAS, a domain-agnostic replication framework built around CLI coding agents. Given a paper, a code repository, or both, VERITAS extracts the paper's claims, runs the methodology while resolving issues as they arise, and judges each claim against the evidence from experiment runs. The pipeline returns an importance-weighted Replication Score, a severity-rated log of every fix applied, and the patched codebase. We evaluate VERITAS on CORE-Bench and ReplicationBench, 65 papers spanning computer science, social science, medicine, and astrophysics. Against two strong Claude Code baselines on the same model and host environment, VERITAS achieves state-of-the-art performance and leads on every metric on both benchmarks.