mlsanityData to 7 Oct 2026

An Image is Worth a Thousand Toxic Words: A Metamorphic Testing Framework for Content Moderation Software

arXiv 2308.09810 ↗First posted 18 Aug 2023cs.SE cs.AI cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

The exponential growth of social media platforms has brought about a revolution in communication and content dissemination in human society. Nevertheless, these platforms are being increasingly misused to spread toxic content, including hate speech, malicious advertising, and pornography, leading to severe negative consequences such as harm to teenagers' mental health. Despite tremendous efforts in developing and deploying textual and image content moderation methods, malicious users can evade moderation by embedding texts into images, such as screenshots of the text, usually with some interference. We find that modern content moderation software's performance against such malicious inputs remains underexplored. In this work, we propose OASIS, a metamorphic testing framework for content moderation software. OASIS employs 21 transform rules summarized from our pilot study on 5,000 real-world toxic contents collected from 4 popular social media applications, including Twitter, Instagram, Sina Weibo, and Baidu Tieba. Given toxic textual contents, OASIS can generate image test cases, which preserve the toxicity yet are likely to bypass moderation. In the evaluation, we employ OASIS to test five commercial textual content moderation software from famous companies (i.e., Google Cloud, Microsoft Azure, Baidu Cloud, Alibaba Cloud and Tencent Cloud), as well as a state-of-the-art moderation research model. The results show that OASIS achieves up to 100% error finding rates. Moreover, through retraining the models with the test cases generated by OASIS, the robustness of the moderation model can be improved without performance degradation.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 44 of its 90 references that are in the map. First posted on arXiv 18 Aug 2023.

Builds on

44 of 90 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  7. 2016Adversarial examples in the physical worldcited by 1,762
  8. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  9. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  10. 2016Universal adversarial perturbationscited by 774
  11. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  12. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
32 more
  1. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  2. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  3. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  4. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 487
  5. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 403
  6. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  7. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  8. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  9. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
  10. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 249
  11. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 215
  12. 2017Deep Learning for Hate Speech Detection in Tweetscited by 180
  13. 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 144
  14. 2022Improving Adversarial Transferability via Neuron Attribution-Based Attackscited by 92
  15. 2020Metamorphic Testing: A New Approach for Generating Next Test Casescited by 71
  16. 2018DLFuzz: Differential Fuzzing Testing of Deep Learning Systemscited by 64
  17. 2023BiasAsker: Measuring the Bias in Conversational AI Systemcited by 52
  18. 2021Hatemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-based Hatecited by 50
  19. 2023Improving the Transferability of Adversarial Samples by Path-Augmented Methodcited by 36
  20. 2019Tackling Online Abuse: A Survey of Automated Abuse Detection Methodscited by 33
  21. 2018Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testingcited by 30
  22. 2021RobOT: Robustness-Oriented Testing for Deep Learning Systemscited by 27
  23. 2023MTTM: Metamorphic Testing for Textual Content Moderation Softwarecited by 20
  24. 2020Testing Machine Translation via Referential Transparencycited by 19
  25. 2022Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translationcited by 14
  26. 2022Tencent's Multilingual Machine Translation System for WMT22 Large-Scale African Languagescited by 13
  27. 2018Fooling OCR Systems with Adversarial Text Imagescited by 12
  28. 2018Mind Your Language: Abuse and Offense Detection for Code-Switched Languagescited by 11
  29. 2020Attacking Optical Character Recognition (OCR) Systems with Adversarial Watermarkscited by 7
  30. 2023Validating Multimedia Content Moderation Software via Semantic Fusioncited by 5
  31. 2018FPETS : Fully Parallel End-to-End Text-to-Speech Systemcited by 4
  32. 2022A Black-Box Attack on Optical Character Recognition Systemscited by 3

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024Efficient Generation of Targeted and Transferable Adversarial Examples for Vision-Language Models Via Diffusion Modelscited by 36
  2. 2024Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detectioncited by 27
  3. 2024New Job, New Gender? Measuring the Social Bias in Image Generation Modelscited by 15