mlsanityData to 14 Oct 2026

Adversarial Arena: Crowdsourcing Data Generation through Interactive Competition

arXiv 2604.17803 ↗First posted 20 Apr 2026cs.AI cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Post-training Large Language Models requires diverse, high-quality data which is rare and costly to obtain, especially in low resource domains and for multi-turn conversations. Common solutions are crowdsourcing or synthetic generation, but both often yield low-quality or low-diversity data. We introduce Adversarial Arena for building high quality conversational datasets by framing data generation as an adversarial task: attackers create prompts, and defenders generate responses. This interactive competition between multiple teams naturally produces diverse and complex data. We validated this approach by conducting a competition with 10 academic teams from top US and European universities, each building attacker or defender bots. The competition, focused on safety alignment of LLMs in cybersecurity, generated 19,683 multi-turn conversations. Fine-tuning an open-source model on this dataset produced an 18.47% improvement in secure code generation on CyberSecEval-Instruct and 29.42% improvement on CyberSecEval-MITRE.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 46 of its 71 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

46 of 71 references.

References · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  6. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  10. 2023Textbooks Are All You Needcited by 569
  11. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  12. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
34 more
  1. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  3. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  4. 2021A Survey of Human-in-the-loop for Machine Learningcited by 215
  5. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 211
  6. 2022Is GPT-3 a Good Data Annotator?cited by 208
  7. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  8. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
  9. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  10. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 187
  11. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
  12. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 138
  13. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 125
  14. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  15. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 100
  16. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 95
  17. 2023Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extractioncited by 89
  18. 2023Automated Annotation with Generative AI Requires Validationcited by 76
  19. 2024Instruction Tuning for Secure Code Generationcited by 68
  20. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  21. 2024The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Modelscited by 63
  22. 2023When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviourcited by 62
  23. 2023Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Modelscited by 61
  24. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  25. 2024Comprehensive Exploration of Synthetic Data Generation: A Surveycited by 60
  26. 2023Curated LLM: Synergy of LLMs and Data Curation for tabular augmentation in low-data regimescited by 52
  27. 2022Self-Prompting Large Language Models for Zero-Shot Open-Domain QAcited by 39
  28. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  29. 2024Generative AI for Synthetic Data Generation: Methods, Challenges and the Futurecited by 33
  30. 2025PurpCode: Reasoning for Safer Code Generationcited by 18
  31. 2024Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Modelscited by 13
  32. 2023Generating Data for Symbolic Language with Large Language Modelscited by 12
  33. 2025RedCoder: Automated Multi-Turn Red Teaming for Code LLMscited by 9
  34. 2025ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistantscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Safe-FCL: Continual Risk Awareness and Fine-Grained Safety Aggregation for Federated Continual Learning of LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts
  2. 2024Adversarial Negotiation Dynamics in Generative Language Models
  3. 2025RainbowPlus: Enhancing Adversarial Prompt Generation via Evolutionary Quality-Diversity Search
  4. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI
  5. 2026The Anatomy of Conversational Scams: A Topic-Based Red Teaming Analysis of Multi-Turn Interactions in LLMs
  6. 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectives
  7. 2025Bot Wars Evolved: Orchestrating Competing LLMs in a Counterstrike Against Phone Scams
  8. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition