mlsanityData to 14 Oct 2026

Automated Harmfulness Testing for Code Large Language Models

arXiv 2503.16740 ↗First posted 20 Mar 2025cs.SEcited by 0

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Generative AI systems powered by Large Language Models (LLMs) usually use content moderation to prevent harmful content spread. To evaluate the robustness of content moderation, several metamorphic testing techniques have been proposed to test content moderation software. However, these techniques mainly focus on general users (e.g., text and image generation). Meanwhile, a recent study shows that developers consider using harmful keywords when naming software artifacts to be an unethical behavior. Exposure to harmful content in software artifacts can negatively impact the mental health of developers, making content moderation for Code Large Language Models (Code LLMs) essential. We conduct a preliminary study on program transformations that can be misused to introduce harmful content into auto-generated code, identifying 32 such transformations. To address this, we propose CHT, a coverage-guided harmfulness testing framework that generates prompts using diverse transformations and harmful keywords injected into benign programs. CHT evaluates output damage to assess potential risks in LLM-generated explanations and code. Our evaluation of four Code LLMs and GPT-4o-mini reveals that content moderation in LLM-based code generation is easily bypassed. To enhance moderation, we propose a two-phase approach that first detects harmful content before generating output, improving moderation effectiveness by 483.76%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 38 of its 76 references that are in the map. First posted on arXiv 20 Mar 2025.

Builds on

38 of 76 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation23 May
  2. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion12 Mar
  3. 2024The Ethics of Interaction: Mitigating Security Threats in LLMs22 Jan

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  4. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  5. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  6. 2022Red Teaming Language Models with Language Modelscited by 1,026
  7. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  8. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  9. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  10. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  11. 2020Recipes for Safety in Open-domain Chatbotscited by 170
  12. 2024CodeGemma: Open Code Models Based on Gemmacited by 150
23 more
  1. 2017Fairness Testing: Testing Software for Discriminationcited by 126
  2. 2021Bias in Machine Learning Software: Why? How? What to do?cited by 93
  3. 2018Automated Directed Fairness Testingcited by 78
  4. 2022Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Modelscited by 68
  5. 2022Fairness Testing: A Comprehensive Survey and Analysis of Trendscited by 62
  6. 2024Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perceptioncited by 56
  7. 2023BiasAsker: Measuring the Bias in Conversational AI Systemcited by 53
  8. 2023ChatGPT and Software Testing Education: Promises & Perilscited by 42
  9. 2023Refactoring Programs Using Large Language Models with Few-Shot Examplescited by 28
  10. 2021BiasFinder: Metamorphic Test Generation to Uncover Bias for Sentiment Analysis Systemscited by 24
  11. 2023MTTM: Metamorphic Testing for Textual Content Moderation Softwarecited by 22
  12. 2024Breaking Bias, Building Bridges: Evaluation and Mitigation of Social Biases in LLMs via Contact Hypothesiscited by 20
  13. 2019Testing DNN Image Classifiers for Confusion & Bias Errorscited by 18
  14. 2021BiasRV: Uncovering Biased Sentiment Predictions at Runtimecited by 12
  15. 2024How to Refactor this Code? An Exploratory Study on Developer-ChatGPT Refactoring Conversationscited by 12
  16. 2024Evaluation Ethics of LLMs in Legal Domaincited by 9
  17. 2024Identifying Inaccurate Descriptions in LLM-generated Code Comments via Test Executioncited by 8
  18. 2023Software Testing of Generative AI Systems: Challenges and Opportunitiescited by 7
  19. 2023Implementing Responsible AI: Tensions and Trade-Offs Between Ethics Aspectscited by 6
  20. 2023Exposing Algorithmic Discrimination and Its Consequences in Modern Society: Insights from a Scoping Studycited by 4
  21. 2024Moving beyond Deletions: Program Simplification via Diverse Program Transformationscited by 1
  22. 2024An Empirical Study of Refactoring Engine Bugscited by 1
  23. 2025Testing Refactoring Engine via Historical Bug Report driven LLMcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024RMCBench: Benchmarking Large Language Models' Resistance to Malicious Code
  2. 2025Code Red! On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Tasks
  3. 2024ShieldGemma: Generative AI Content Moderation Based on Gemma
  4. 2025ASTRAL: Automated Safety Testing of Large Language Models
  5. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLM
  6. 2024S-Eval: Towards Automated and Comprehensive Safety Evaluation for Large Language Models
  7. 2025Can LLMs Rank the Harmfulness of Smaller LLMs? We are Not There Yet
  8. 2026Bidirectional Empowerment of Metamorphic Testing and Large Language Models: A Systematic Survey