mlsanityData to 13 Oct 2026

BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems

arXiv 2606.20668 ↗First posted 12 Jun 2026cs.CR cs.AI cs.LGcited by 1

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

LLM supervision systems, namely input/output moderation filters and jailbreak detectors, are the primary safeguard against misuse in deployed AI applications, yet existing benchmarks are often vendor-biased, omit cost and latency, and rarely compare specialized guardrails against repurposed generalist LLMs. We present BELLS-O (Benchmark for the Evaluation of LLM Supervision Systems, Operational), the first independent operational benchmark of LLM supervision systems. BELLS-O evaluates 28 systems from 17 providers: every major specialized guardrail (e.g., LlamaGuard-4, ShieldGemma-2, Lakera Guard) and frontier generalists repurposed as supervisors (e.g., GPT-5.4, Claude Sonnet 4.6, Grok-4.1), jointly on detection rate, false-positive rate, latency, and monetary cost. We cover input/output moderation across 11 harm categories and jailbreak detection across 13 attack techniques, using in-house datasets built from handcrafted prompts, expert-curated samples, and quality-controlled synthetic generation. To suppress latent generator fingerprints in synthetic data, every generated sample is paraphrased. Mapping the Pareto frontier reveals use-case-dependent tradeoffs. On content moderation, specialized supervisors are operationally dominant: top systems match frontier LLMs on detection (~95% vs. 94%) at comparably low false-positive rates (<=2%), while running 5-10x faster and ~10x cheaper. On jailbreak detection, the tradeoff shifts: frontier LLMs achieve higher detection and lower false-positive rates but at 10-50x higher cost and 5-10x higher latency. We release the benchmark, framework, leaderboard, and datasets as the first vendor-neutral basis for selecting safeguards under real deployment constraints.

Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 24 of its 41 references that are in the map. First posted on arXiv 12 Jun 2026.

Builds on

24 of 41 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs22 Jul
  2. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
  3. 2024A StrongREJECT for Empty Jailbreaks15 Feb
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  5. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 18

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,107
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,405
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,141
  5. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 784
  6. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 354
  7. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 261
  8. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 238
  9. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 222
  10. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 182
  11. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 128
  12. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 112
6 more
  1. 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 72
  2. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 59
  3. 2024Your Large Language Models Are Leaving Fingerprintscited by 32
  4. 2024BELLS: A Framework Towards Future Proof Benchmarks for the Evaluation of LLM Safeguardscited by 14
  5. 2025Natural Fingerprints of Large Language Modelscited by 6
  6. 2025An Example Safety Case for Safeguards Against Misusecited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Hard-Gate Candidacy in a Deployed Validator Suitecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The bitter lesson of misuse detection
  2. 2026Gate AI: LLM Security Benchmark Evaluation Methodology and Results
  3. 2026When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift
  4. 2025Adversarial Prompt Evaluation: Systematic Benchmarking of Guardrails Against Prompt Input Attacks on LLMs
  5. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs
  6. 2026LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments
  7. 2026Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Models
  8. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defenses