TestAug: A Framework for Augmenting Capability-based NLP Tests
Not on the map. Why?
Abstract (arXiv)
The recently proposed capability-based NLP testing allows model developers to test the functional capabilities of NLP models, revealing functional failures that cannot be detected by the traditional heldout mechanism. However, existing work on capability-based testing requires extensive manual efforts and domain expertise in creating the test cases. In this paper, we investigate a low-cost approach for the test case generation by leveraging the GPT-3 engine. We further propose to use a classifier to remove the invalid outputs from GPT-3 and expand the outputs into templates to generate more test cases. Our experiments show that TestAug has three advantages over the existing work on behavioral testing: (1) TestAug can find more bugs than existing work; (2) The test cases in TestAug are more diverse; and (3) TestAug largely saves the manual efforts in creating the test suites. The code and data for TestAug can be found at our project website (https://guanqun-yang.github.io/testaug/) and GitHub (https://github.com/guanqun-yang/testaug).
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 14 Oct 2022.
Builds on
29 of 38 references.
References · 29
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
- 2015A large annotated corpus for learning natural language inferencecited by 1,847
- 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
- 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
- 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
- 2018Annotation Artifacts in Natural Language Inference Datacited by 671
- 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
- 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
- 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
17 more
- 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 218
- 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 216
- 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
- 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
- 2021Generating Datasets with Pretrained Language Modelscited by 193
- 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 175
- 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
- 2021A Recipe For Arbitrary Text Style Transfer with Large Language Modelscited by 107
- 2020Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESuppositioncited by 77
- 2019Probing Natural Language Inference Models through Semantic Fragmentscited by 64
- 2021Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-based Hatecited by 50
- 2020Why Attentions May Not Be Interpretable?cited by 41
- 2019HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoningcited by 35
- 2020TaxiNLI: Taking a Ride up the NLU Hillcited by 27
- 2021As Easy as 1, 2, 3: Behavioural Testing of NMT Systems for Numerical Translationcited by 8
- 2021What Do You See in this Patient? Behavioral Testing of Clinical NLP Modelscited by 6
- 2021LoNLI: An Extensible Framework for Testing Diverse Logical Reasoning Capabilities for NLIcited by 3
Built on by
Not cited yet.