# MTTM: Metamorphic Testing for Textual Content Moderation Software

arXiv: https://arxiv.org/abs/2302.05706 · HTML: https://mlsanity.com/paper/2302.05706/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 87 references that are in the map.
- First posted on arXiv 11 Feb 2023.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://mlsanity.com/paper/2004.09984/) (2020)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers](https://mlsanity.com/paper/1801.04354/) (2018)
- [BAE: BERT-based Adversarial Examples for Text Classification](https://mlsanity.com/paper/2004.01970/) (2020)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Deep Learning for Hate Speech Detection in Tweets](https://mlsanity.com/paper/1706.00188/) (2017)
- [Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systems](https://mlsanity.com/paper/1903.11508/) (2019)
- [Improving Adversarial Transferability via Neuron Attribution-Based Attacks](https://mlsanity.com/paper/2204.00008/) (2022)
- [Reevaluating Adversarial Examples in Natural Language](https://mlsanity.com/paper/2004.14174/) (2020)
- [Metamorphic Testing: A New Approach for Generating Next Test Cases](https://mlsanity.com/paper/2002.12543/) (2020)
- [Bad Characters: Imperceptible NLP Attacks](https://mlsanity.com/paper/2106.09898/) (2021)
- [Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-based Hate](https://mlsanity.com/paper/2108.05921/) (2021)
- [Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving](https://mlsanity.com/paper/2101.06784/) (2021)
- [Automatic Testing and Improvement of Machine Translation](https://mlsanity.com/paper/1910.02688/) (2019)
- [Tackling Online Abuse: A Survey of Automated Abuse Detection Methods](https://mlsanity.com/paper/1908.06024/) (2019)
- [Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing](https://mlsanity.com/paper/1808.05353/) (2018)
- [Structure-Invariant Testing for Machine Translation](https://mlsanity.com/paper/1907.08710/) (2019)
- [RobOT: Robustness-Oriented Testing for Deep Learning Systems](https://mlsanity.com/paper/2102.05913/) (2021)
- [Testing Machine Translation via Referential Transparency](https://mlsanity.com/paper/2004.10361/) (2020)
- [Coverage Guided Testing for Recurrent Neural Networks](https://mlsanity.com/paper/1911.01952/) (2019)

## Built on by (most cited first, 22 of 22)

- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench](https://mlsanity.com/paper/2308.03656/) (2023)
- [BiasAsker: Measuring the Bias in Conversational AI System](https://mlsanity.com/paper/2305.12434/) (2023)
- [Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection](https://mlsanity.com/paper/2404.09894/) (2024)
- [METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities](https://mlsanity.com/paper/2312.06056/) (2023)
- [New Job, New Gender? Measuring the Social Bias in Image Generation Models](https://mlsanity.com/paper/2401.00763/) (2024)
- [Metamorphic Testing of Large Language Models for Natural Language Processing](https://mlsanity.com/paper/2511.02108/) (2025)
- ["Oops, Did I Just Say That?" Testing and Repairing Unethical Suggestions of Large Language Models with Suggest-Critique-Reflect Process](https://mlsanity.com/paper/2305.02626/) (2023)
- [MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systems](https://mlsanity.com/paper/2412.15557/) (2024)
- [Validating Multimedia Content Moderation Software via Semantic Fusion](https://mlsanity.com/paper/2305.13623/) (2023)
- [MeTMaP: Metamorphic Testing for Detecting False Vector Matching Problems in LLM Augmented Generation](https://mlsanity.com/paper/2402.14480/) (2024)
- [An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software](https://mlsanity.com/paper/2308.09810/) (2023)
- [An Empirical Study on Large Language Models in Accuracy and Robustness under Chinese Industrial Scenarios](https://mlsanity.com/paper/2402.01723/) (2024)
- [Are All Prompt Components Value-Neutral? Understanding the Heterogeneous Adversarial Robustness of Dissected Prompt in Large Language Models](https://mlsanity.com/paper/2508.01554/) (2025)
- [Fairness Testing in Retrieval-Augmented Generation: How Small Perturbations Reveal Bias in Small Language Models](https://mlsanity.com/paper/2509.26584/) (2025)
- [VEglue: Testing Visual Entailment Systems via Object-Aligned Joint Erasing](https://mlsanity.com/paper/2403.02581/) (2024)
- [Automated Harmfulness Testing for Code Large Language Models](https://mlsanity.com/paper/2503.16740/) (2025)
- [MT4DP: Data Poisoning Attack Detection for DL-based Code Search Models via Metamorphic Testing](https://mlsanity.com/paper/2507.11092/) (2025)
- [Metamorphic Testing for Audio Content Moderation Software](https://mlsanity.com/paper/2509.24215/) (2025)
- [Chinese Morph Resolution in E-commerce Live Streaming Scenarios](https://mlsanity.com/paper/2512.23280/) (2025)
- [Think Before You Code: Dual Reasoning for the NLSafety-Utility Trade-Off in LLM Code Generation](https://mlsanity.com/paper/2604.12088/) (2026)
- [The Consistency Dilemma in LLMs: Generator-Evaluator Agreement and Vulnerability to Mistakes](https://mlsanity.com/paper/2606.30653/) (2026)

## Abstract (arXiv)

The exponential growth of social media platforms such as Twitter and Facebook has revolutionized textual communication and textual content publication in human society. However, they have been increasingly exploited to propagate toxic content, such as hate speech, malicious advertisement, and pornography, which can lead to highly negative impacts (e.g., harmful effects on teen mental health). Researchers and practitioners have been enthusiastically developing and extensively deploying textual content moderation software to address this problem. However, we find that malicious users can evade moderation by changing only a few words in the toxic content. Moreover, modern content moderation software performance against malicious inputs remains underexplored. To this end, we propose MTTM, a Metamorphic Testing framework for Textual content Moderation software. Specifically, we conduct a pilot study on 2,000 text messages collected from real users and summarize eleven metamorphic relations across three perturbation levels: character, word, and sentence. MTTM employs these metamorphic relations on toxic textual contents to generate test cases, which are still toxic yet likely to evade moderation. In our evaluation, we employ MTTM to test three commercial textual content moderation software and two state-of-the-art moderation algorithms against three kinds of toxic content. The results show that MTTM achieves up to 83.9%, 51%, and 82.5% error finding rates (EFR) when testing commercial moderation software provided by Google, Baidu, and Huawei, respectively, and it obtains up to 91.2% EFR when testing the state-of-the-art algorithms from the academy. In addition, we leverage the test cases generated by MTTM to retrain the model we explored, which largely improves model robustness (0% to 5.9% EFR) while maintaining the accuracy on the original test set.