An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice
Not on the map. Why?
Abstract (arXiv)
Claims about AI safety reach audiences well beyond the AI community, yet many rely on opaque evidence or static assessments, when supporting evidence is accessible at all. We present the Systemic Risk Index, an open evaluation pipeline and dashboard built to make empirical evidence more transparent and traceable to the public. Our work organizes 19 public benchmarks into four systemic-risk categories defined by the EU GPAI Code of Practice---CBRN, cyber offense, harmful manipulation, and loss of control---and evaluates models using harm-preserving perturbations and simulated deployment contexts. The interactive dashboard lets users alternate between average and worst-case aggregation, vary how model capability affects the aggregate score, and trace each risk rating to its benchmark evidence. Across 18 models, scores fall by 14 to 37 points under worst-case aggregation, highlighting information that can be hidden by an average assessment of model risk. LLM judges show agreement with human graders comparable to human--human agreement (kappa = 0.78--0.82), and a blind audit finds that 83% of sampled transformations preserve the original harm. In a survey (N = 21), most participants report that scores are easy to understand and that the dashboard encouraged them to view model evaluations under different settings
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 23 Sep 2026.
Builds on
18 of 26 references.
References · 18
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
- 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
- 2018Model Cards for Model Reportingcited by 1,068
- 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
- 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 380
- 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
- 2019Measurement and Fairnesscited by 299
- 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
6 more
- 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
- 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
- 2024The AI risk repository: A meta-review, database, and taxonomy of risks from artificial intelligencecited by 13
- 2025Bench-2-CoP: Can We Trust Benchmarking for EU AI Compliance?cited by 6
- 2025SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requestscited by 5
- 2026AI Security Leaderboard: Methodology, Results and Minimal Standardcited by 1
Built on by
Not cited yet.