Frontier AI Regulation: Managing Emerging Risks to Public Safety
Not on the map. Why?
Abstract (arXiv)
Advanced AI models hold the promise of tremendous benefits for humanity, but society needs to proactively manage the accompanying risks. In this paper, we focus on what we term "frontier AI" models: highly capable foundation models that could possess dangerous capabilities sufficient to pose severe risks to public safety. Frontier AI models pose a distinct regulatory challenge: dangerous capabilities can arise unexpectedly; it is difficult to robustly prevent a deployed model from being misused; and, it is difficult to stop a model's capabilities from proliferating broadly. To address these challenges, at least three building blocks for the regulation of frontier models are needed: (1) standard-setting processes to identify appropriate requirements for frontier AI developers, (2) registration and reporting requirements to provide regulators with visibility into frontier AI development processes, and (3) mechanisms to ensure compliance with safety standards for the development and deployment of frontier AI models. Industry self-regulation is an important first step. However, wider societal discussions and government intervention will be needed to create standards and to ensure compliance with them. We consider several options to this end, including granting enforcement powers to supervisory authorities and licensure regimes for frontier AI models. Finally, we propose an initial set of safety standards. These include conducting pre-deployment risk assessments; external scrutiny of model behavior; using risk assessments to inform deployment decisions; and monitoring and responding to new information about model capabilities and uses post-deployment. We hope this discussion contributes to the broader conversation on how to balance public safety risks and innovation benefits from advances at the frontier of AI development.
Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 96 of its 249 references that are in the map. First posted on arXiv 6 Jul 2023.
Builds on
96 of 249 references.
References · 96
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
38 more
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2023A Survey of Large Language Modelscited by 2,537
- 2022Emergent Abilities of Large Language Modelscited by 2,483
- 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
- 2022Survey of Hallucination in Natural Language Generationcited by 2,056
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
- 2022Large Language Models Encode Clinical Knowledgecited by 1,807
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
- 2016Concrete Problems in AI Safetycited by 1,582
- 2018Datasheets for Datasetscited by 1,371
- 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
- 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
- 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
- 2018Model Cards for Model Reportingcited by 1,068
- 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
- 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
- 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
- 2022Red Teaming Language Models with Language Modelscited by 983
- 2022In-context Learning and Induction Headscited by 930
- 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
- 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
- 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
- 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
- 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
- 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
- 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
- 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
- 2019Release Strategies and the Social Impacts of Language Modelscited by 505
- 2018AI safety via debatecited by 384
- 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 380
- 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
- 2023Large Language Models as Tool Makerscited by 293
- 2021Unsolved Problems in ML Safetycited by 291
- 2019Tackling Climate Change with Machine Learningcited by 264
- 2023The False Promise of Imitating Proprietary LLMscited by 254
- 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
- 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
50 most cited of 96 All 96 (JSON)
Built on by
Citing papers per year: 2023: 17; 2024: 36; 2025: 24; 2026: 18.
95 papers.
Sort:
By area:
- 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
- 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
- 2023Can LLM-Generated Misinformation Be Detected?cited by 173
- 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
- 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
- 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
- 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
- 2024Visibility into AI Agentscited by 77
- 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 77
- 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 69
- 2024On the Societal Impact of Open Foundation Modelscited by 64
- 2023Managing extreme AI risks amid rapid progresscited by 48
38 more
- 2023Welfare Diplomacy: Benchmarking Language Model Cooperationcited by 43
- 2023Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Modelscited by 40
- 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integrationcited by 38
- 2024Are large language models superhuman chemists?cited by 34
- 2023Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectivescited by 31
- 2024Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementationcited by 27
- 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
- 2024Can Editing LLMs Inject Harm?cited by 22
- 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 21
- 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
- 2023Position: Key Claims in LLM Research Have a Long Tail of Footnotescited by 20
- 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
- 2024Mapping the Ethics of Generative AI: A Comprehensive Scoping Reviewcited by 19
- 2023Market Concentration Implications of Foundation Modelscited by 15
- 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
- 2024Foundation Model Transparency Reportscited by 14
- 2024Training Compute Thresholds: Features and Functions in AI Regulationcited by 13
- 2024Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agentscited by 13
- 2023A Systematic Survey of Prompt Engineering on Vision-Language Foundation Modelscited by 12
- 2023Towards Publicly Accountable Frontier LLMs: Building an External Scrutiny Ecosystem under the ASPIRE Frameworkcited by 11
- 2024Evaluating AI Evaluation: Perils and Prospectscited by 11
- 2025Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agentscited by 11
- 2024Responsible Reporting for Frontier AI Developmentcited by 10
- 2024Societal Adaptation to Advanced AIcited by 10
- 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitationscited by 10
- 2024Trust AI Regulation? Discerning users are vital to build trust and effective AI regulationcited by 9
- 2024The Necessity of AI Audit Standards Boardscited by 9
- 2023International Governance of Civilian AI: A Jurisdictional Certification Approachcited by 8
- 2024An Auditing Test To Detect Behavioral Shift in Language Modelscited by 8
- 2026Quantifying Trust: Financial Risk Management for Trustworthy AI Agentscited by 8
- 2023Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Modelscited by 7
- 2024Exploring ChatGPT and its Impact on Societycited by 6
- 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
- 2023Towards Responsible Governance of Biological Design Toolscited by 5
- 2024XLM for Autonomous Driving Systems: A Comprehensive Reviewcited by 5
- 2025AI Governance through Marketscited by 5
- 2025Real-World Gaps in AI Governance Researchcited by 5
- 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
50 most cited of 95 All 95 (JSON)
- 2026Silent Revision: Measuring Undisclosed Change in the Safety Frameworks of Frontier AI Developers8 Sep 2026
- 2026The Normalization of Deviance in AI Development4 Sep 2026
- 2026FUSE: An Evaluating Framework for Dangerous Capabilities of LLMs2 Sep 2026
- 2026Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attacks26 Aug 2026
- 2026SciHazard: A Benchmark for Measuring Scientific Safety Risks with Decomposed Harm Scoring21 Jul 2026
- 2026Physiological Noise Augmentation Improves Non-Invasive Brain-to-Speech6 Jul 2026
- 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap29 Jun 2026
- 2026The Age of AI Agents Demands A New Scientific Paradigm To Sustain Trustworthy Science18 Jun 2026
- 2026AI Researchers Must Help Lead Arms Control to Mitigate Military AI Risks10 Jun 2026
- 2026NeurIPS Should Require Reproducibility Standards for Frontier AI Safety Claims5 May 2026
- 2026Artificial Jagged Intelligence as Uneven Optimization Energy Allocation Capability Concentration, Redistribution, and Optimization Governance2 May 2026
- 2026To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems30 Apr 2026
38 more
- 2026Quantifying Trust: Financial Risk Management for Trustworthy AI Agents5 Apr 2026
- 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook26 Mar 2026
- 2026Alignment Is Not Enough: A Relational Framework for Moral Standing in Human-AI Interaction13 Feb 2026
- 2026The Limits of AI Data Transparency Policy: Three Disclosure Fallacies26 Jan 2026
- 2026AI as Entertainment13 Jan 2026
- 2026Computational Compliance for AI Regulation: Blueprint for a New Research Domain8 Jan 2026
- 2025Causal-Guided Detoxify Backdoor Attack of Open-Weight LoRA Models22 Dec 2025
- 2025Provably Safe Model Updates1 Dec 2025
- 2025PolicyPad: Collaborative Prototyping of LLM Policies24 Sep 2025
- 2025SAM Encoder Breach by Adversarial Simplicial Complex Triggers Downstream Model Failures8 Aug 2025
- 2025Deciding how to respond: A deliberative framework to guide policymaker responses to AI systems5 Aug 2025
- 2025From Turing to Tomorrow: The UK's Approach to AI Regulation3 Jul 2025
- 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safety17 Jun 2025
- 2025Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priors29 May 2025
- 2025What Is AI Safety? What Do We Want It to Be?5 May 2025
- 2025The Precautionary Principle and the Innovation Principle: Incompatible Guides for AI Innovation Governance?1 May 2025
- 2025Real-World Gaps in AI Governance Research30 Apr 2025
- 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment22 Apr 2025
- 2025Trends in Frontier AI Model Count: A Forecast to 202821 Apr 2025
- 2025Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents11 Apr 2025
- 2025Frontier AI's Impact on the Cybersecurity Landscape7 Apr 2025
- 2025What Makes an Evaluation Useful? Common Pitfalls and Best Practices30 Mar 2025
- 2025The Backfiring Effect of Weak AI Safety Regulation26 Mar 2025
- 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration20 Mar 2025
- 2025The AI Pentad, the CHARME2D Model, and an Assessment of Current-State AI Regulation8 Mar 2025
- 2025Quantifying Security Vulnerabilities: A Metric-Driven Security Analysis of Gaps in Current AI Standards12 Feb 2025
- 2025Position: It's Time to Act on the Risk of Efficient Personalized Text Generation10 Feb 2025
- 2025AI Governance through Markets29 Jan 2025
- 2025Human services organizations and the responsible integration of AI: Considering ethics and contextualizing risk(s)20 Jan 2025
- 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions8 Jan 2025
- 2024Towards Data Governance of Frontier AI Models5 Dec 2024
- 2024Declare and Justify: Explicit assumptions in AI evaluations are necessary for effective regulation19 Nov 2024
- 2024Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems30 Oct 2024
- 2024Standardization Trends on Safety and Trustworthiness Technology for Advanced AI29 Oct 2024
- 2024An Auditing Test To Detect Behavioral Shift in Language Models25 Oct 2024
- 2024Ethics Whitepaper: Whitepaper on Ethical Research into Large Language Models17 Oct 2024
- 2024SAGE: Scalable Ground Truth Evaluations for Large Sparse Autoencoders9 Oct 2024
- 2024Data-Centric AI Governance: Addressing the Limitations of Model-Focused Policies25 Sep 2024
50 newest of 95 All 95 (JSON)
- 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
- 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 77
- 2024Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agentscited by 13
- 2025Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agentscited by 11
- 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
- 2026The Age of AI Agents Demands A New Scientific Paradigm To Sustain Trustworthy Sciencecited by 1
- 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap29 Jun 2026
- 2026The Age of AI Agents Demands A New Scientific Paradigm To Sustain Trustworthy Science18 Jun 2026
- 2025Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents11 Apr 2025
- 2024Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agents5 Aug 2024
- 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies28 Jul 2024
- 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stages16 Feb 2024
- 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integrationcited by 38
- 2024Can Editing LLMs Inject Harm?cited by 22
- 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
- 2026Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attackscited by 0
- 2026Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attacks26 Aug 2026
- 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safety17 Jun 2025
- 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration20 Mar 2025
- 2024Can Editing LLMs Inject Harm?29 Jul 2024
- 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
- 2024Standardization Trends on Safety and Trustworthiness Technology for Advanced AIcited by 0
- 2025Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priorscited by 0
- 2025Neither Stochastic Parroting nor AGI: LLMs Solve Tasks through Context-Directed Extrapolation from Training Data Priors29 May 2025
- 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment22 Apr 2025
- 2024Standardization Trends on Safety and Trustworthiness Technology for Advanced AI29 Oct 2024
- 2024XLM for Autonomous Driving Systems: A Comprehensive Review16 Sep 2024
- 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
- 2023Can LLM-Generated Misinformation Be Detected?cited by 173
- 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
- 2024Visibility into AI Agentscited by 77
- 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 69
- 2024On the Societal Impact of Open Foundation Modelscited by 64
- 2023Managing extreme AI risks amid rapid progresscited by 48
- 2023Welfare Diplomacy: Benchmarking Language Model Cooperationcited by 43
- 2023Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Modelscited by 40
- 2024Are large language models superhuman chemists?cited by 34
- 2023Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectivescited by 31
- 2024Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementationcited by 27
12 most cited of 76 All 76 (JSON)
- 2026Silent Revision: Measuring Undisclosed Change in the Safety Frameworks of Frontier AI Developers8 Sep 2026
- 2026The Normalization of Deviance in AI Development4 Sep 2026
- 2026FUSE: An Evaluating Framework for Dangerous Capabilities of LLMs2 Sep 2026
- 2026SciHazard: A Benchmark for Measuring Scientific Safety Risks with Decomposed Harm Scoring21 Jul 2026
- 2026Physiological Noise Augmentation Improves Non-Invasive Brain-to-Speech6 Jul 2026
- 2026AI Researchers Must Help Lead Arms Control to Mitigate Military AI Risks10 Jun 2026
- 2026NeurIPS Should Require Reproducibility Standards for Frontier AI Safety Claims5 May 2026
- 2026Artificial Jagged Intelligence as Uneven Optimization Energy Allocation Capability Concentration, Redistribution, and Optimization Governance2 May 2026
- 2026To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems30 Apr 2026
- 2026Quantifying Trust: Financial Risk Management for Trustworthy AI Agents5 Apr 2026
- 2026Alignment Is Not Enough: A Relational Framework for Moral Standing in Human-AI Interaction13 Feb 2026
- 2026The Limits of AI Data Transparency Policy: Three Disclosure Fallacies26 Jan 2026
12 newest of 76 All 76 (JSON)