# SmartOracle -- An Agentic Approach to Mitigate Noise in Differential Oracles

arXiv: https://arxiv.org/abs/2601.15074 · HTML: https://mlsanity.com/paper/2601.15074/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 58 references that are in the map.
- First posted on arXiv 21 Jan 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 15 of 15)

- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Small Language Models are the Future of Agentic AI](https://mlsanity.com/paper/2506.02153/) (2025)
- [Agents in Software Engineering: Survey, Landscape, and Vision](https://mlsanity.com/paper/2409.09030/) (2024)
- [Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectives](https://mlsanity.com/paper/2310.01152/) (2023)
- [Evaluating Fuzz Testing](https://mlsanity.com/paper/1808.09700/) (2018)
- [Multi-role Consensus through LLMs Discussions for Vulnerability Detection](https://mlsanity.com/paper/2403.14274/) (2024)
- [Automated Conformance Testing for JavaScript Engines via Deep Compiler Fuzzing](https://mlsanity.com/paper/2104.07460/) (2021)
- [Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation](https://mlsanity.com/paper/2506.02943/) (2025)
- [Fundamentals of Building Autonomous LLM Agents](https://mlsanity.com/paper/2510.09244/) (2025)
- [FRUGAL: Unlocking SSL for Software Analytics](https://mlsanity.com/paper/2108.09847/) (2021)
- [Testing Database Engines via Pivoted Query Synthesis](https://mlsanity.com/paper/2001.04174/) (2020)
- [JEST: N+1-version Differential Testing of Both JavaScript Engines and Specification](https://mlsanity.com/paper/2102.07498/) (2021)
- [Exposing Bugs in JavaScript Engines through Test Transplantation and Differential Testing](https://mlsanity.com/paper/2012.03759/) (2020)
- [FalseCrashReducer: Mitigating False Positive Crashes in OSS-Fuzz-Gen Using Agentic AI](https://mlsanity.com/paper/2510.02185/) (2025)
- [When Less is More: On the Value of "Co-training" for Semi-Supervised Software Defect Predictors](https://mlsanity.com/paper/2211.05920/) (2022)

## Abstract (arXiv)

Differential fuzzers detect bugs by executing identical inputs across distinct implementations of the same specification, such as JavaScript interpreters. Validating the outputs requires an oracle and for differential testing of JavaScript, these are constructed manually, making them expensive, time-consuming, and prone to false positives. Worse, when the specification evolves, this manual effort must be repeated. Inspired by the success of agentic systems in other SE domains, this paper introduces SmartOracle. SmartOracle decomposes the manual triage workflow into specialized Large Language Model (LLM) sub-agents. These agents synthesize independently gathered evidence from terminal runs and targeted specification queries to reach a final verdict. For historical benchmarks, SmartOracle achieves 0.84 recall with an 18% false positive rate. Compared to a sequential Gemini 2.5 Pro baseline, it improves triage accuracy while reducing analysis time by 4× and API costs by 10×. In active fuzzing campaigns, SmartOracle successfully identified and reported previously unknown specification-level issues across major engines, including bugs in V8, JavaScriptCore, and GraalJS. The success of SmartOracle's agentic architecture on Javascript suggests it might be useful other software systems- a research direction we will explore in future work.