mlsanityData to 14 Oct 2026

Query-Efficient Black-Box Red Teaming via Bayesian Optimization

arXiv 2305.17444 ↗First posted 27 May 2023cs.AI cs.CL cs.CRcited by 32

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

The deployment of large-scale generative models is often restricted by their potential risk of causing harm to users in unpredictable ways. We focus on the problem of black-box red teaming, where a red team generates test cases and interacts with the victim model to discover a diverse set of failures with limited query access. Existing red teaming methods construct test cases based on human supervision or language model (LM) and query all test cases in a brute-force manner without incorporating any information from past evaluations, resulting in a prohibitively large number of queries. To this end, we propose Bayesian red teaming (BRT), novel query-efficient black-box red teaming methods based on Bayesian optimization, which iteratively identify diverse positive test cases leading to model failures by utilizing the pre-defined user input pool and the past evaluations. Experimental results on various user input pools demonstrate that our method consistently finds a significantly larger number of diverse positive test cases under the limited query budget than the baseline methods. The source code is available at https://github.com/snu-mllab/Bayesian-Red-Teaming.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 36 of its 68 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

36 of 68 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
23 more
  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  3. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  4. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  5. 2018A Tutorial on Bayesian Optimizationcited by 744
  6. 2022Pseudo Numerical Methods for Diffusion Models on Manifoldscited by 700
  7. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  8. 2016Stealing Machine Learning Models via Prediction APIscited by 619
  9. 2020Recipes for building an open-domain chatbotcited by 593
  10. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  11. 2022Red-Teaming the Stable Diffusion Safety Filtercited by 264
  12. 2020HateCheck: Functional Tests for Hate Speech Detection Modelscited by 216
  13. 2019The Second Conversational Intelligence Challenge (ConvAI2)cited by 179
  14. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  15. 2020Recipes for Safety in Open-domain Chatbotscited by 170
  16. 2018Counterfactual Fairness in Text Classification through Robustnesscited by 151
  17. 2017ParlAI: A Dialog Research Software Platformcited by 139
  18. 2021Improving Question Answering Model Robustness with Synthetic Adversarial Data Generationcited by 79
  19. 2021Towards Improving Adversarial Training of NLP Modelscited by 79
  20. 2022GODEL: Large-Scale Pre-Training for Goal-Directed Dialogcited by 51
  21. 2016Batched Gaussian Process Bandit Optimization via Determinantal Point Processescited by 33
  22. 2022Detection of Word Adversarial Examples in Text Classification: Benchmark and Baseline via Robust Density Estimationcited by 32
  23. 2022Query-Efficient and Scalable Black-Box Adversarial Attacks on Discrete Sequential Data via Bayesian Optimizationcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 13; 2025: 6; 2026: 6.

32 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 254
  2. 2023Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?cited by 250
  3. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  4. 2023Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Promptscited by 171
  5. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 110
  6. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 88
  7. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 83
  8. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  9. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  10. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  11. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 37
  12. 2023Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attackscited by 35
20 more
  1. 2024Gradient-Based Language Model Red Teamingcited by 34
  2. 2024Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interactioncited by 25
  3. 2023Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbotscited by 15
  4. 2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 13
  5. 2024Efficient Black-box Adversarial Attacks via Bayesian Optimization Guided by a Function Priorcited by 11
  6. 2025RainbowPlus: Enhancing Adversarial Prompt Generation via Evolutionary Quality-Diversity Searchcited by 11
  7. 2025Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Modelscited by 9
  8. 2024Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intentcited by 8
  9. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  10. 2024Can LLMs be Fooled? Investigating Vulnerabilities in LLMscited by 5
  11. 2025Text-Diffusion Red-Teaming of Large Language Models: Unveiling Harmful Behaviors with Proximity Constraintscited by 5
  12. 2026Adaptive auditing of AI systems with anytime-valid guaranteescited by 3
  13. 2026TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Modelscited by 2
  14. 2026ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluationcited by 2
  15. 2024Red-Teaming Text-to-Image Models via In-Context Experience Replay and Semantic-Preserving Prompt Rewritingcited by 1
  16. 2025RedRFT: A Light-Weight Benchmark for Reinforcement Fine-Tuning-Based Red Teamingcited by 1
  17. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  18. 2025AutoODD: Agentic Audits via Bayesian Red Teaming in Black-Box Modelscited by 0
  19. 2026Generating Attacks for LLMs with GFlowNetscited by 0
  20. 2026Narrative Paradoxes in LLM Inference: Shaping Reasoning Trajectories toward Misalignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Against The Achilles' Heel: A Survey on Red Teaming for Generative Models
  2. 2026CART: Closed-Loop Adaptive Red Teaming for Large Language Models
  3. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratch
  4. 2025DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modeling
  5. 2025Be a Multitude to Itself: A Prompt Evolution Framework for Red Teaming
  6. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
  7. 2025SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning
  8. 2025Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models