# Reliability Testing for Natural Language Processing Systems

arXiv: https://arxiv.org/abs/2105.02590 · HTML: https://mlsanity.com/paper/2105.02590/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 53 of its 129 references that are in the map.
- First posted on arXiv 6 May 2021.

## Builds on (most cited first, 30 of 53)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://mlsanity.com/paper/2004.09984/) (2020)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [BAE: BERT-based Adversarial Examples for Text Classification](https://mlsanity.com/paper/2004.01970/) (2020)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://mlsanity.com/paper/2004.06100/) (2020)
- [Adversarial Example Generation with Syntactically Controlled Paraphrase Networks](https://mlsanity.com/paper/1804.06059/) (2018)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [A Survey of the State of Explainable AI for Natural Language Processing](https://mlsanity.com/paper/2010.00711/) (2020)
- [Neural Text Summarization: A Critical Evaluation](https://mlsanity.com/paper/1908.08960/) (2019)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)

## Built on by (most cited first, 25 of 25)

- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [Measure and Improve Robustness in NLP Models: A Survey](https://mlsanity.com/paper/2112.08313/) (2021)
- [On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents](https://mlsanity.com/paper/2408.00989/) (2024)
- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation](https://mlsanity.com/paper/2112.02721/) (2021)
- [Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models](https://mlsanity.com/paper/2206.08325/) (2022)
- [Automatic Construction of Evaluation Suites for Natural Language Generation Datasets](https://mlsanity.com/paper/2106.09069/) (2021)
- [The Birth of Bias: A case study on the evolution of gender bias in an English language model](https://mlsanity.com/paper/2207.10245/) (2022)
- ["One-Size-Fits-All"? Examining Expectations around What Constitute "Fair" or "Good" NLG System Behaviors](https://mlsanity.com/paper/2310.15398/) (2023)
- [Chaos with Keywords: Exposing Large Language Models Sycophantic Hallucination to Misleading Keywords and Evaluating Defense Strategies](https://mlsanity.com/paper/2406.03827/) (2024)
- [TAI3: Testing Agent Integrity in Interpreting User Intent](https://mlsanity.com/paper/2506.07524/) (2025)
- [LEAP: Efficient and Automated Test Method for NLP Software](https://mlsanity.com/paper/2308.11284/) (2023)
- [The Risks of Machine Learning Systems](https://mlsanity.com/paper/2204.09852/) (2022)
- [TraVLR: Now You See It, Now You Don't! A Bimodal Dataset for Evaluating Visio-Linguistic Reasoning](https://mlsanity.com/paper/2111.10756/) (2021)
- [MeTMaP: Metamorphic Testing for Detecting False Vector Matching Problems in LLM Augmented Generation](https://mlsanity.com/paper/2402.14480/) (2024)
- [Language Invariant Properties in Natural Language Processing](https://mlsanity.com/paper/2109.13037/) (2021)
- [Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework](https://mlsanity.com/paper/2505.03563/) (2025)
- [Risks and NLP Design: A Case Study on Procedural Document QA](https://mlsanity.com/paper/2408.11860/) (2024)
- [BotSIM: An End-to-End Bot Simulation Framework for Commercial Task-Oriented Dialog Systems](https://mlsanity.com/paper/2211.11982/) (2022)
- [Ethics Whitepaper: Whitepaper on Ethical Research into Large Language Models](https://mlsanity.com/paper/2410.19812/) (2024)
- [Reliability of Topic Modeling](https://mlsanity.com/paper/2410.23186/) (2024)
- [Exploring Robustness of LLMs to Paraphrasing Based on Sociodemographic Factors](https://mlsanity.com/paper/2501.08276/) (2025)
- [BotSIM: An End-to-End Bot Simulation Toolkit for Commercial Task-Oriented Dialog Systems](https://mlsanity.com/paper/2211.15916/) (2022)
- [A Group-Specific Approach to NLP for Hate Speech Detection](https://mlsanity.com/paper/2304.11223/) (2023)
- [AutoTestForge: A Multidimensional Automated Testing Framework for Natural Language Processing Models](https://mlsanity.com/paper/2503.05102/) (2025)

## Abstract (arXiv)

Questions of fairness, robustness, and transparency are paramount to address before deploying NLP systems. Central to these concerns is the question of reliability: Can NLP systems reliably treat different demographics fairly and function correctly in diverse and noisy environments? To address this, we argue for the need for reliability testing and contextualize it among existing work on improving accountability. We show how adversarial attacks can be reframed for this goal, via a framework for developing reliability tests. We argue that reliability testing -- with an emphasis on interdisciplinary collaboration -- will enable rigorous and targeted testing, and aid in the enactment and enforcement of industry standards.