Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems
Not on the map. Why?
Abstract (arXiv)
There is an urgent need to identify both short and long-term risks from newly emerging types of Artificial Intelligence (AI), as well as available risk management measures. In response, and to support global efforts in regulating AI and writing safety standards, we compile an extensive catalog of risk sources and risk management measures for general-purpose AI (GPAI) systems, complete with descriptions and supporting examples where relevant. This work involves identifying technical, operational, and societal risks across model development, training, and deployment stages, as well as surveying established and experimental methods for managing these risks. To the best of our knowledge, this paper is the first of its kind to provide extensive documentation of both GPAI risk sources and risk management measures that are descriptive, self-contained and neutral with respect to any existing regulatory framework. This work intends to help AI providers, standards experts, researchers, policymakers, and regulators in identifying and mitigating systemic risks from GPAI systems. For this reason, the catalog is released under a public domain license for ease of direct use by stakeholders in AI governance and standards.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 121 of its 238 references that are in the map. First posted on arXiv 30 Oct 2024.
Builds on
121 of 238 references.
References · 121
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2013Intriguing properties of neural networkscited by 5,155
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
- 2022Emergent Abilities of Large Language Modelscited by 2,512
38 more
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
- 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
- 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
- 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
- 2022Mass-Editing Memory in a Transformercited by 1,001
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2022Quantifying Memorization Across Neural Language Modelscited by 763
- 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
- 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
- 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
- 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
- 2019Release Strategies and the Social Impacts of Language Modelscited by 513
- 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
- 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
- 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
- 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
- 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
- 2018AI safety via debatecited by 400
- 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
- 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 348
- 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
- 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
- 2023Low-Resource Languages Jailbreak GPT-4cited by 309
- 2018Certified Defenses against Adversarial Examplescited by 306
- 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
- 2021Unsolved Problems in ML Safetycited by 292
- 2023Faithful Chain-of-Thought Reasoningcited by 288
- 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
- 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
- 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
- 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 226
- 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
- 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
- 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 221
- 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
50 most cited of 121 All 121 (JSON)
Built on by
Citing papers per year: 2025: 2.
2 papers.