mlsanityData to 7 Oct 2026

LEAP: Efficient and Automated Test Method for NLP Software

arXiv 2308.11284 ↗First posted 22 Aug 2023cs.SE cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

The widespread adoption of DNNs in NLP software has highlighted the need for robustness. Researchers proposed various automatic testing techniques for adversarial test cases. However, existing methods suffer from two limitations: weak error-discovering capabilities, with success rates ranging from 0% to 24.6% for BERT-based NLP software, and time inefficiency, taking 177.8s to 205.28s per test case, making them challenging for time-constrained scenarios. To address these issues, this paper proposes LEAP, an automated test method that uses LEvy flight-based Adaptive Particle swarm optimization integrated with textual features to generate adversarial test cases. Specifically, we adopt Levy flight for population initialization to increase the diversity of generated test cases. We also design an inertial weight adaptive update operator to improve the efficiency of LEAP's global optimization of high-dimensional text examples and a mutation operator based on the greedy strategy to reduce the search time. We conducted a series of experiments to validate LEAP's ability to test NLP software and found that the average success rate of LEAP in generating adversarial test cases is 79.1%, which is 6.1% higher than the next best approach (PSOattack). While ensuring high success rates, LEAP significantly reduces time overhead by up to 147.6s compared to other heuristic-based methods. Additionally, the experimental results demonstrate that LEAP can generate more transferable test cases and significantly enhance the robustness of DNN-based systems.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 38 of its 76 references that are in the map. First posted on arXiv 22 Aug 2023.

Builds on

38 of 76 references.

References · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  10. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  11. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  12. 2018Generating Natural Language Adversarial Examplescited by 422
26 more
  1. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  2. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 162
  3. 2016Recurrent Neural Network for Text Classification with Multi-Task Learningcited by 144
  4. 2020Contextualized Perturbation for Textual Adversarial Attackcited by 136
  5. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  6. 2017Deep Text Classification Can be Fooledcited by 105
  7. 2019Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systemscited by 99
  8. 2018On Adversarial Examples for Character-Level Neural Machine Translationcited by 85
  9. 2021Towards Improving Adversarial Training of NLP Modelscited by 79
  10. 2021Language Model Evaluation Beyond Perplexitycited by 77
  11. 2022Natural Attack for Pre-trained Models of Codecited by 72
  12. 2018DLFuzz: Differential Fuzzing Testing of Deep Learning Systemscited by 64
  13. 2020Deep Learning for Source Code Modeling and Generation: Models, Applications and Challengescited by 55
  14. 2021Meta Gradient Adversarial Attackcited by 55
  15. 2020Investigating Societal Biases in a Poetry Composition Systemcited by 47
  16. 2018Robust Machine Comprehension Models via Adversarial Trainingcited by 44
  17. 2020CAT-Gen: Improving Robustness in NLP Models via Controlled Adversarial Text Generationcited by 43
  18. 2021Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Modelscited by 29
  19. 2020Augmented Lagrangian Adversarial Attackscited by 28
  20. 2021Grey-box Adversarial Attack And Defence For Sentiment Classificationcited by 28
  21. 2021Reliability Testing for Natural Language Processing Systemscited by 25
  22. 2022Query-Efficient and Scalable Black-Box Adversarial Attacks on Discrete Sequential Data via Bayesian Optimizationcited by 22
  23. 2017FairFuzz: Targeting Rare Branches to Rapidly Increase Greybox Fuzz Testing Coveragecited by 20
  24. 2020On the Transferability of Adversarial Attacksagainst Neural Text Classifiercited by 16
  25. 2022AEON: A Method for Automatic Evaluation of NLP Test Casescited by 15
  26. 2019TERMINATOR: Better Automated UI Test Case Prioritizationcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 3.

6 papers.

  1. 2024VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulationcited by 43
  2. 2024MeTMaP: Metamorphic Testing for Detecting False Vector Matching Problems in LLM Augmented Generationcited by 4
  3. 2025No Query, No Accesscited by 4
  4. 2024RITFIS: Robust input testing framework for LLMs-based intelligent softwarecited by 3
  5. 2025Bounded PCTL Model Checking of Large Language Model Outputscited by 1
  6. 2025AutoTestForge: A Multidimensional Automated Testing Framework for Natural Language Processing Modelscited by 0