# Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis arXiv: https://arxiv.org/abs/2510.26423 ยท HTML: https://mlsanity.com/paper/2510.26423/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 31 of its 52 references that are in the map. - First posted on arXiv 30 Oct 2025. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 30 of 31) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021) - [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024) - [An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation](https://mlsanity.com/paper/2302.06527/) (2023) - [Unit Test Case Generation with Transformers and Focal Context](https://mlsanity.com/paper/2009.05617/) (2020) - [Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM](https://mlsanity.com/paper/2402.00097/) (2024) - [EffiBench: Benchmarking the Efficiency of Automatically Generated Code](https://mlsanity.com/paper/2402.02037/) (2024) - [TOGA: A Neural Method for Test Oracle Generation](https://mlsanity.com/paper/2109.09262/) (2021) - [Can Large Language Models Transform Natural Language Intent into Formal Method Postconditions?](https://mlsanity.com/paper/2310.01831/) (2023) - [Using Large Language Models to Generate JUnit Tests: An Empirical Study](https://mlsanity.com/paper/2305.00418/) (2023) - [Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing](https://mlsanity.com/paper/2308.16557/) (2023) - [TESTEVAL: Benchmarking Large Language Models for Test Case Generation](https://mlsanity.com/paper/2406.04531/) (2024) - [HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation](https://mlsanity.com/paper/2412.21199/) (2024) - [CoverUp: Effective High Coverage Test Generation for Python](https://mlsanity.com/paper/2403.16218/) (2024) - [HITS: High-coverage LLM-based Unit Test Generation via Method Slicing](https://mlsanity.com/paper/2408.11324/) (2024) - [A3Test: Assertion-Augmented Automated Test Case Generation](https://mlsanity.com/paper/2302.10352/) (2023) - [CAT-LM: Training Language Models on Aligned Code And Tests](https://mlsanity.com/paper/2310.01602/) (2023) - [EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code](https://mlsanity.com/paper/2505.13004/) (2025) - [TOGLL: Correct and Strong Test Oracle Generation with LLMs](https://mlsanity.com/paper/2405.03786/) (2024) - [Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation](https://mlsanity.com/paper/2310.02368/) (2023) - [TestART: Improving LLM-based Unit Testing via Co-evolution of Automated Generation and Repair Iteration](https://mlsanity.com/paper/2408.03095/) (2024) - [Advancing Code Coverage: Incorporating Program Analysis with Large Language Models](https://mlsanity.com/paper/2404.04966/) (2024) - [Hallucination to Consensus: Multi-Agent LLMs for End-to-End JUnit Test Generation](https://mlsanity.com/paper/2506.02943/) (2025) - [Benchmarking LLMs for Unit Test Generation from Real-World Functions](https://mlsanity.com/paper/2508.00408/) (2025) - [Domain Adaptation for Code Model-based Unit Test Case Generation](https://mlsanity.com/paper/2308.08033/) (2023) - [LLM-based Unit Test Generation via Property Retrieval](https://mlsanity.com/paper/2410.13542/) (2024) - [Large Language Models for Unit Testing: A Systematic Literature Review](https://mlsanity.com/paper/2506.15227/) (2025) - [UniTSyn: A Large-Scale Dataset Capable of Enhancing the Prowess of Large Language Models for Program Testing](https://mlsanity.com/paper/2402.03396/) (2024) - [CasModaTest: A Cascaded and Model-agnostic Self-directed Framework for Unit Test Generation](https://mlsanity.com/paper/2406.15743/) (2024) - [Improving Deep Assertion Generation via Fine-Tuning Retrieval-Augmented Pre-trained Language Models](https://mlsanity.com/paper/2502.16071/) (2025) ## Built on by (most cited first, 4 of 4) - [LogicHunter: Testing LLM Agent Frameworks with an Agentic Oracle](https://mlsanity.com/paper/2607.06195/) (2026) - [MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models](https://mlsanity.com/paper/2606.24815/) (2026) - [Paper Espresso: From Paper Overload to Research Insight](https://mlsanity.com/paper/2604.04562/) (2026) - [Continuous Process-Level Evaluation for Evolving Enterprise AI Agent Skills](https://mlsanity.com/paper/2610.01833/) (2026) ## Abstract (arXiv) Test oracle generation in non-regression testing is a longstanding challenge in software engineering, where the goal is to produce oracles that can accurately determine whether a function under test (FUT) behaves as intended for a given input. In this paper, we introduce Nexus, a novel multi-agent framework to address this challenge. Nexus generates test oracles by leveraging a diverse set of specialized agents that synthesize test oracles through a structured process of deliberation, validation, and iterative self-refinement. During the deliberation phase, a panel of four specialist agents, each embodying a distinct testing philosophy, collaboratively critiques and refines an initial set of test oracles. Then, in the validation phase, Nexus generates a plausible candidate implementation of the FUT and executes the proposed oracles against it in a secure sandbox. For any oracle that fails this execution-based check, Nexus activates an automated selfrefinement loop, using the specific runtime error to debug and correct the oracle before re-validation. Our extensive evaluation on seven diverse benchmarks demonstrates that Nexus consistently and substantially outperforms state-of-theart baselines. For instance, Nexus improves the test-level oracle accuracy on the LiveCodeBench from 46.30% to 57.73% for GPT-4.1-Mini. The improved accuracy also significantly enhances downstream tasks: the bug detection rate of GPT4.1-Mini generated test oracles on HumanEval increases from 90.91% to 95.45% for Nexus compared to baselines, and the success rate of automated program repair improves from 35.23% to 69.32%.