mlsanityData to 7 Oct 2026

AEON: A Method for Automatic Evaluation of NLP Test Cases

arXiv 2205.06439 ↗First posted 13 May 2022cs.SE cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Due to the labor-intensive nature of manual test oracle construction, various automated testing techniques have been proposed to enhance the reliability of Natural Language Processing (NLP) software. In theory, these techniques mutate an existing test case (e.g., a sentence with its label) and assume the generated one preserves an equivalent or similar semantic meaning and thus, the same label. However, in practice, many of the generated test cases fail to preserve similar semantic meaning and are unnatural (e.g., grammar errors), which leads to a high false alarm rate and unnatural test cases. Our evaluation study finds that 44% of the test cases generated by the state-of-the-art (SOTA) approaches are false alarms. These test cases require extensive manual checking effort, and instead of improving NLP software, they can even degrade NLP software when utilized in model training. To address this problem, we propose AEON for Automatic Evaluation Of NLP test cases. For each generated test case, it outputs scores based on semantic similarity and language naturalness. We employ AEON to evaluate test cases generated by four popular testing techniques on five datasets across three typical NLP tasks. The results show that AEON aligns the best with human judgment. In particular, AEON achieves the best average precision in detecting semantic inconsistent test cases, outperforming the best baseline metric by 10%. In addition, AEON also has the highest average precision of finding unnatural test cases, surpassing the baselines by more than 15%. Moreover, model training with test cases prioritized by AEON leads to models that are more accurate and robust, demonstrating AEON's potential in improving NLP software.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 51 of its 90 references that are in the map. First posted on arXiv 13 May 2022.

Builds on

51 of 90 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
38 more
  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  4. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  5. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  6. 2017Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networkscited by 484
  7. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  8. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  9. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  10. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
  11. 2020Evaluation of Text Generation: A Surveycited by 275
  12. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 249
  13. 2018code2vec: Learning Distributed Representations of Codecited by 249
  14. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 213
  15. 2019Certified Robustness to Adversarial Word Substitutionscited by 164
  16. 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 144
  17. 2018Generating 3D Adversarial Point Cloudscited by 144
  18. 2020Contextualized Perturbation for Textual Adversarial Attackcited by 136
  19. 2018Guiding Deep Learning System Testing using Surprise Adequacycited by 123
  20. 2017Deep Text Classification Can be Fooledcited by 105
  21. 2022Improving Adversarial Transferability via Neuron Attribution-Based Attackscited by 92
  22. 2020Evolution of Semantic Similarity -- A Surveycited by 83
  23. 2020Reevaluating Adversarial Examples in Natural Languagecited by 83
  24. 2018DeepMutation: Mutation Testing of Deep Learning Systemscited by 66
  25. 2018DeepBugs: A Learning Approach to Name-based Bug Detectioncited by 65
  26. 2020Robust Encodings: A Framework for Combating Adversarial Typoscited by 58
  27. 2019On Evaluation of Adversarial Perturbations for Sequence-to-Sequence Modelscited by 56
  28. 2018Adversarial Sample Detection for Deep Neural Network through Model Mutation Testingcited by 48
  29. 2018Attacks Meet Interpretability: Attribute-steered Detection of Adversarial Samplescited by 38
  30. 2020Unsupervised Text Generation by Learning from Searchcited by 36
  31. 2019Automatic Testing and Improvement of Machine Translationcited by 35
  32. 2019SirenAttack: Generating Adversarial Audio for End-to-End Acoustic Systemscited by 33
  33. 2020Discern: Discourse-Aware Entailment Reasoning Network for Conversational Machine Readingcited by 33
  34. 2019Structure-Invariant Testing for Machine Translationcited by 27
  35. 2019Towards Structured Evaluation of Deep Neural Network Supervisorscited by 19
  36. 2020Testing Machine Translation via Referential Transparencycited by 19
  37. 2019Coverage Guided Testing for Recurrent Neural Networkscited by 15
  38. 2021Towards Variable-Length Textual Adversarial Attackscited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 3; 2025: 2; 2026: 1.

15 papers.

Sort:

  1. 2023Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBenchcited by 54
  2. 2023BiasAsker: Measuring the Bias in Conversational AI Systemcited by 52
  3. 2023Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularizationcited by 42
  4. 2023Improving the Transferability of Adversarial Samples by Path-Augmented Methodcited by 36
  5. 2023MTTM: Metamorphic Testing for Textual Content Moderation Softwarecited by 20
  6. 2023Towards Reasonable Budget Allocation in Untargeted Graph Structure Attacks via Gradient Debiascited by 16
  7. 2024New Job, New Gender? Measuring the Social Bias in Image Generation Modelscited by 15
  8. 2025Metamorphic Testing of Large Language Models for Natural Language Processingcited by 12
  9. 2023LEAP: Efficient and Automated Test Method for NLP Softwarecited by 6
  10. 2023Validating Multimedia Content Moderation Software via Semantic Fusioncited by 5
  11. 2023LUNA: A Model-Based Universal Analysis Framework for Large Language Modelscited by 4
  12. 2024Machine Translation Testing via Syntactic Tree Pruningcited by 2
3 more
  1. 2024Computer Vision Intelligence Test Modeling and Generation: A Case Study on Smart OCRcited by 2
  2. 2025Metamorphic Testing for Audio Content Moderation Softwarecited by 0
  3. 2026SemLink: A Semantic-Aware Automated Test Oracle for Hyperlink Verification using Siamese Sentence-BERTcited by 0