mlsanityData to 7 Oct 2026

Automatic Hard Example Synthesis with Multi-Level Agentic Data Curation

arXiv 2607.14256 ↗First posted 15 Jul 2026cs.AI cs.MAcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Multimodal Large Language Models (MLLMs) are increasingly deployed for nuanced content safety and moderation tasks, yet they remain vulnerable to adversarial attacks and out-of-distribution edge cases. Traditional active learning and manual annotation fail to scale against the complexity and volume of novel multimodal threats. In this paper, we propose an automated, agentic red-teaming framework that systematically synthesizes difficult examples using an iterative strategy that proposes novel hypotheses as well as mutating on past attempts. Leveraging a multi-agent architecture that consists of a high-reasoning Architect agent, an advanced image generator, and a multi-level verification committee of LLM raters, our system autonomously uncovers boundary-pushing violations and ambiguous policy edge cases without any human intervention. By employing these carefully synthesized adversarial examples as in-context demonstrations via test-time Retrieval, we substantially improve the target model's robustness, reducing the False Negative Rate (FNR) from 41.2% to 24.5% in a public image safety benchmark without relying on any human labeling.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 52 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

50 of 52 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Every Picture Tells a Dangerous Story: Memory-Augmented Multi-Agent Jailbreak Attacks on VLMs14 Apr
  2. 2026Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Models14 Apr
  3. 2026Adversarial Attacks on Multimodal Large Language Models: A Comprehensive Survey30 Mar
  4. 2026Evolving Contextual Safety in Multi-Modal Large Language Models via Inference-Time Self-Reflective Memory16 Mar
  5. 2026The Side Effects of Being Smart: Safety Risks in MLLMs' Multi-Image Reasoning20 Jan
  6. 2026Adversarial Defense in Vision-Language Models: An Overview18 Jan
  7. 2026MTMCS-Bench: Evaluating Contextual Safety of Multimodal Large Language Models in Multi-Turn Dialogues11 Jan
  8. 2025OutSafe-Bench: A Benchmark for Multimodal Offensive Content Detection in Large Language Models13 Nov
  9. 2025AI vs. Human Moderators: A Comparative Evaluation of Multimodal LLMs in Content Moderation for Brand Safety7 Aug
  10. 2025HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Model5 Jun

Other references · 40

  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2018AI safety via debatecited by 384
  4. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  5. 2024LlavaGuard: An Open VLM-based Framework for Safeguarding Vision Datasets and Modelscited by 49
  6. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 42
  7. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  8. 2023Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledgecited by 32
  9. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  10. 2023Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understandingcited by 28
  11. 2025Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoningcited by 28
  12. 2025Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoningcited by 27
28 more
  1. 2025MOSAIC: Modeling Social AI for Content Dissemination and Regulation in Multi-Agent Simulationscited by 26
  2. 2024Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QAcited by 25
  3. 2026Benchmark Test-Time Scaling of General LLM Agentscited by 24
  4. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMscited by 16
  5. 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidancecited by 15
  6. 2024Debate, Deliberate, Decide (D3): A Cost-Aware Adversarial Framework for Reliable and Interpretable LLM Evaluationcited by 14
  7. 2026BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluationcited by 12
  8. 2025AI Debate Aids Assessment of Controversial Claimscited by 9
  9. 2025ContextNav: Towards Agentic Multimodal In-Context Learningcited by 9
  10. 2025UniGame: Turning a Unified Multimodal Model Into Its Own Adversarycited by 9
  11. 2026THINKSAFE: Self-Generated Safety Alignment for Reasoning Modelscited by 9
  12. 2025Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMscited by 7
  13. 2026Llama-3.1-FoundationAI-SecurityLLM-Reasoning-8B Technical Reportcited by 7
  14. 2026T-MAP: Red-Teaming LLM Agents with Trajectory-aware Evolutionary Searchcited by 6
  15. 2025Who Judges the Judge? LLM Jury-on-Demand: Building Trustworthy LLM Evaluation Systemscited by 5
  16. 2024Supporting Human Raters with the Detection of Harmful Content using Large Language Modelscited by 4
  17. 2025A Judge-free LLM Open-ended Generation Benchmark Based on the Distributional Hypothesiscited by 4
  18. 2026Visual Exclusivity Attacks: Automatic Multimodal Red Teaming via Agentic Planningcited by 4
  19. 2026Courtroom-Style Multi-Agent Debate with Progressive RAG and Role-Switching for Controversial Claim Verificationcited by 4
  20. 2026Security in LLM-as-a-Judge: A Comprehensive SoKcited by 4
  21. 2025ELHPlan: Efficient Long-Horizon Task Planning for Multi-Agent Collaborationcited by 3
  22. 2026TSQueryBench: LLM-as-a-Judge for Time Series Explanationscited by 3
  23. 2025Aetheria: A multimodal interpretable content safety framework based on multi-agent debate and collaborationcited by 2
  24. 2026Jailbreaking LLMs & VLMs: Mechanisms, Evaluation, and Unified Defensecited by 2
  25. 2026Evaluating Nova 2.0 Lite model under Amazon's Frontier Model Safety Frameworkcited by 2
  26. 2026DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detectioncited by 2
  27. 2025Curriculum Guided Massive Multi Agent System Solving For Robust Long Horizon Taskscited by 1
  28. 2026Automated Concept Discovery for LLM-as-a-Judge Preference Analysiscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Yuvion VL: A Multimodal Foundation Model for Adversarial Content and AI Safety
  2. 2026Automatic Red Teaming for Implicit Vulnerabilities of Text-to-Image Models
  3. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs
  4. 2024Dissecting Adversarial Robustness of Multimodal LM Agents
  5. 2026CourtGuard: A Model-Agnostic Framework for Zero-Shot Policy Adaptation in LLM Safety
  6. 2025Agentic Moderation: Multi-Agent Design for Safer Vision-Language Models
  7. 2025Automating Steering for Safe Multimodal Large Language Models
  8. 2026ReFrame: Evidence-Guided Test-Time Safety Alignment in Multimodal Large Language Models