# ToxiCraft: A Novel Framework for Synthetic Generation of Harmful Information

arXiv: https://arxiv.org/abs/2409.14740 · HTML: https://mlsanity.com/paper/2409.14740/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 32 of its 55 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation](https://mlsanity.com/paper/2104.08826/) (2021)
- [Data Augmentation for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/1705.00440/) (2017)
- [Toxicity Detection: Does Context Really Matter?](https://mlsanity.com/paper/2006.00998/) (2020)
- [Reasoning Implicit Sentiment with Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.11255/) (2023)
- [Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method](https://mlsanity.com/paper/2305.13412/) (2023)
- [ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback](https://mlsanity.com/paper/2210.12329/) (2022)
- [PromDA: Prompt-based Data Augmentation for Low-Resource NLU Tasks](https://mlsanity.com/paper/2202.12499/) (2022)
- [Deep Learning Models for Multilingual Hate Speech Detection](https://mlsanity.com/paper/2004.06465/) (2020)
- [Data Boost: Text Data Augmentation Through Reinforcement Learning Guided Conditional Generation](https://mlsanity.com/paper/2012.02952/) (2020)
- [Hate Towards the Political Opponent: A Twitter Corpus Study of the 2020 US Elections on the Basis of Offensive Speech and Stance Detection](https://mlsanity.com/paper/2103.01664/) (2021)
- [Overview of the HASOC track at FIRE 2020: Hate Speech and Offensive Content Identification in Indo-European Languages](https://mlsanity.com/paper/2108.05927/) (2021)
- [Robust Hate Speech Detection in Social Media: A Cross-Dataset Empirical Evaluation](https://mlsanity.com/paper/2307.01680/) (2023)
- [Simulated Chats for Building Dialog Systems: Learning to Generate Conversations from Instructions](https://mlsanity.com/paper/2010.10216/) (2020)
- [People Make Better Edits: Measuring the Efficacy of LLM-Generated Counterfactually Augmented Data for Harmful Language Detection](https://mlsanity.com/paper/2311.01270/) (2023)
- [Mixture of Soft Prompts for Controllable Data Generation](https://mlsanity.com/paper/2303.01580/) (2023)
- [Fight Fire with Fire: Fine-tuning Hate Detectors using Large Samples of Generated Hate Speech](https://mlsanity.com/paper/2109.00591/) (2021)
- [Predictive Embeddings for Hate Speech Detection on Twitter](https://mlsanity.com/paper/1809.10644/) (2018)

## Built on by (most cited first, 17 of 17)

- [Value of Information: A Framework for Human-Agent Communication](https://mlsanity.com/paper/2601.06407/) (2026)
- [When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning](https://mlsanity.com/paper/2502.19158/) (2025)
- [Toward Safe and Human-Aligned Game Conversational Recommendation via Multi-Agent Decomposition](https://mlsanity.com/paper/2504.20094/) (2025)
- [A Survey on Efficient Large Language Model Training: From Data-centric Perspectives](https://mlsanity.com/paper/2510.25817/) (2025)
- [TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law](https://mlsanity.com/paper/2507.21134/) (2025)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [Privacy-R1: Privacy-Aware Multi-LLM Agent Collaboration via Reinforcement Learning](https://mlsanity.com/paper/2510.16054/) (2025)
- [ToxiLab: How Well Do Open-Source LLMs Generate Synthetic Toxicity Data?](https://mlsanity.com/paper/2411.15175/) (2024)
- [Synthetic vs. Gold: The Role of LLM Generated Labels and Data in Cyberbullying Detection](https://mlsanity.com/paper/2502.15860/) (2025)
- [ToxSyn-PT: A Synthetic Fine-Grained Dataset of Minority-Targeted Toxic Language in Portuguese](https://mlsanity.com/paper/2506.10245/) (2025)
- [Steer Model beyond Assistant: Controlling System Prompt Strength via Contrastive Decoding](https://mlsanity.com/paper/2601.06403/) (2026)
- [SynBullying: A Multi LLM Synthetic Conversational Dataset for Cyberbullying Detection](https://mlsanity.com/paper/2511.11599/) (2025)
- [ToxiGAN: Toxic Data Augmentation via LLM-Guided Directional Adversarial Generation](https://mlsanity.com/paper/2601.03121/) (2026)
- [From Measurement Instruments to Data: Leveraging Theory-Driven Synthetic Training Data for Classifying Social Constructs](https://mlsanity.com/paper/2410.12622/) (2024)
- [Beyond Static Benchmarks: Synthesizing Harmful Content via Persona-based Simulation for Robust Evaluation](https://mlsanity.com/paper/2604.17020/) (2026)
- [Lower-Resource, Higher Scores: Language Bias in LLM Evaluators](https://mlsanity.com/paper/2607.14480/) (2026)
- [Do Social Patterns Hold in Synthetic Data? Analyzing Cyberbullying Dynamics in LLM-Generated and Authentic Dialogues](https://mlsanity.com/paper/2609.17549/) (2026)

## Abstract (arXiv)

In different NLP tasks, detecting harmful content is crucial for online environments, especially with the growing influence of social media. However, previous research has two main issues: 1) a lack of data in low-resource settings, and 2) inconsistent definitions and criteria for judging harmful content, requiring classification models to be robust to spurious features and diverse. We propose Toxicraft, a novel framework for synthesizing datasets of harmful information to address these weaknesses. With only a small amount of seed data, our framework can generate a wide variety of synthetic, yet remarkably realistic, examples of toxic information. Experimentation across various datasets showcases a notable enhancement in detection model robustness and adaptability, surpassing or close to the gold labels.