# Lightweight Safety Guardrails via Synthetic Data and RL-guided Adversarial Training

arXiv: https://arxiv.org/abs/2507.08284 · HTML: https://mlsanity.com/paper/2507.08284/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Safety of self-evolving systems, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 11 Jul 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Safety of self-evolving systems](https://mlsanity.com/synth/safety-of-self-evolving-systems/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 20 of 20)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations](https://mlsanity.com/paper/2312.06674/) (2023)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs](https://mlsanity.com/paper/2406.18495/) (2024)
- [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://mlsanity.com/paper/2310.06987/) (2023)
- [GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher](https://mlsanity.com/paper/2308.06463/) (2023)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [A Holistic Approach to Undesired Content Detection in the Real World](https://mlsanity.com/paper/2208.03274/) (2022)
- [SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models](https://mlsanity.com/paper/2402.05044/) (2024)
- [HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models](https://mlsanity.com/paper/2410.01524/) (2024)
- [A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs](https://mlsanity.com/paper/2410.18779/) (2024)
- [FreeAL: Towards Human-Free Active Learning in the Era of Large Language Models](https://mlsanity.com/paper/2311.15614/) (2023)
- [Fine-tuning Language Models with Generative Adversarial Reward Modelling](https://mlsanity.com/paper/2305.06176/) (2023)
- [Learning by Semantic Similarity Makes Abstractive Summarization Better](https://mlsanity.com/paper/2002.07767/) (2020)
- [Deep Domain Adaptation: A Sim2Real Neural Approach for Improving Eye-Tracking Systems](https://mlsanity.com/paper/2403.15947/) (2024)

## Built on by (most cited first, 1 of 1)

- [Fence: Specialized SLM Guardrails for LLM Applications](https://mlsanity.com/paper/2607.18268/) (2026)

## Abstract (arXiv)

We introduce a lightweight yet highly effective safety guardrail framework for language models, demonstrating that small-scale language models can achieve, and even surpass, the performance of larger counterparts in content moderation tasks. This is accomplished through high-fidelity synthetic data generation and adversarial training. The synthetic data generation process begins with human-curated seed data, which undergoes query augmentation and paraphrasing to create diverse and contextually rich examples. This augmented data is then subjected to multiple rounds of curation, ensuring high fidelity and relevance. Inspired by recent advances in the Generative Adversarial Network (GAN) architecture, our adversarial training employs reinforcement learning to guide a generator that produces challenging synthetic examples. These examples are used to fine-tune the safety classifier, enhancing its ability to detect and mitigate harmful content. Additionally, we incorporate strategies from recent research on efficient LLM training, leveraging the capabilities of smaller models to improve the performance of larger generative models. With iterative adversarial training and the generation of diverse, high-quality synthetic data, our framework enables small language models (SLMs) to serve as robust safety guardrails. This approach not only reduces computational overhead but also enhances resilience against adversarial attacks, offering a scalable and efficient solution for content moderation in AI systems.