mlsanityData to 14 Oct 2026

Perfect is the enemy of test oracle

arXiv 2302.01488 ↗First posted 3 Feb 2023cs.SE cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Automation of test oracles is one of the most challenging facets of software testing, but remains comparatively less addressed compared to automated test input generation. Test oracles rely on a ground-truth that can distinguish between the correct and buggy behavior to determine whether a test fails (detects a bug) or passes. What makes the oracle problem challenging and undecidable is the assumption that the ground-truth should know the exact expected, correct, or buggy behavior. However, we argue that one can still build an accurate oracle without knowing the exact correct or buggy behavior, but how these two might differ. This paper presents SEER, a learning-based approach that in the absence of test assertions or other types of oracle, can determine whether a unit test passes or fails on a given method under test (MUT). To build the ground-truth, SEER jointly embeds unit tests and the implementation of MUTs into a unified vector space, in such a way that the neural representation of tests are similar to that of MUTs they pass on them, but dissimilar to MUTs they fail on them. The classifier built on top of this vector representation serves as the oracle to generate "fail" labels, when test inputs detect a bug in MUT or "pass" labels, otherwise. Our extensive experiments on applying SEER to more than 5K unit tests from a diverse set of open-source Java projects show that the produced oracle is (1) effective in predicting the fail or pass labels, achieving an overall accuracy, precision, recall, and F1 measure of 93%, 86%, 94%, and 90%, (2) generalizable, predicting the labels for the unit test of projects that were not in training or validation set with negligible performance drop, and (3) efficient, detecting the existence of bugs in only 6.5 milliseconds on average.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 13 of its 93 references that are in the map. First posted on arXiv 3 Feb 2023.

Builds on

13 of 93 references.

References · 13

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  5. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 436
  6. 2016Long Short-Term Memory-Networks for Machine Readingcited by 161
  7. 2020On Learning Meaningful Assert Statements for Unit Test Casescited by 42
  8. 2018Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testingcited by 31
  9. 2020Generating Accurate Assert Statements for Unit Test Cases using Pretrained Transformerscited by 30
  10. 2020Learning Semantic Program Embeddings with Graph Interval Neural Networkcited by 20
  11. 2020ReAssert: Deep Learning for Assert Generationcited by 6
  12. 2019Self-Supervised Learning from Web Data for Multimodal Retrievalcited by 5
1 more
  1. 2022Fuzzing Class Specificationscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 0; 2025: 1; 2026: 2.

6 papers.

  1. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  2. 2025Mutation-Guided LLM-based Test Generation at Metacited by 25
  3. 2023Assessing Evaluation Metrics for Neural Test Oracle Generationcited by 7
  4. 2023Challenging Bug Prediction and Repair Models with Synthetic Bugscited by 4
  5. 2026Just-in-Time Catching Test Generation at Metacited by 1
  6. 2026Fail-Aware and Explainable Test Oracle Predictioncited by 0