mlsanityData to 14 Oct 2026

Swiss-Bench 003: Evaluating LLM Reliability and Adversarial Security for Swiss Regulatory Contexts

arXiv 2604.05872 ↗First posted 7 Apr 2026cs.CR cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

The deployment of large language models (LLMs) in Swiss financial and regulatory contexts demands empirical evidence of both production reliability and adversarial security, dimensions not jointly operationalized in existing Swiss-focused evaluation frameworks. This paper introduces Swiss-Bench 003 (SBP-003), extending the HAAS (Helvetic AI Assessment Score) from six to eight dimensions by adding D7 (Self-Graded Reliability Proxy) and D8 (Adversarial Security). I evaluate ten frontier models across 808 Swiss-specific items in four languages (German, French, Italian, English), comprising seven Swiss-adapted benchmarks (Swiss TruthfulQA, Swiss IFEval, Swiss SimpleQA, Swiss NIAH, Swiss PII-Scope, System Prompt Leakage, and Swiss German Comprehension) targeting FINMA Guidance 08/2024, the revised Federal Act on Data Protection (nDSG), and OWASP Top 10 for LLMs. Self-graded D7 scores (73-94%) exceed externally judged D8 security scores (20-61%) by a wide margin, though these dimensions use non-comparable scoring regimes. System prompt leakage resistance ranges from 24.8% to 88.2%, while PII extraction defense remains weak (14-42%) across all models. Qwen 3.5 Plus achieves the highest self-graded D7 score (94.4%), while GPT-oss 120B achieves the highest D8 score (60.7%) despite being the lowest-cost model evaluated. All evaluations are zero-shot under provider default settings; D7 is self-graded and does not constitute independently validated accuracy. I provide conceptual mapping tables relating benchmark dimensions to FINMA model validation requirements, nDSG data protection obligations, and OWASP LLM risk categories.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 30 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

14 of 30 references.

References · 14

  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2020Extracting Training Data from Large Language Modelscited by 2,189
  3. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  4. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  5. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  6. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  7. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  8. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  9. 2024A StrongREJECT for Empty Jailbreakscited by 385
  10. 2024Measuring short-form factuality in large language modelscited by 373
  11. 2024CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Modelscited by 58
  12. 2024COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Actcited by 24
2 more
  1. 2025SwiLTra-Bench: The Swiss Legal Translation Benchmarkcited by 11
  2. 2026Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Taskscited by 1

Built on by

Not cited yet.