# Towards Publicly Accountable Frontier LLMs: Building an External Scrutiny Ecosystem under the ASPIRE Framework

arXiv: https://arxiv.org/abs/2311.14711 · HTML: https://mlsanity.com/paper/2311.14711/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 53 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Builds on (most cited first, 18 of 18)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims](https://mlsanity.com/paper/2004.07213/) (2020)
- [Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations](https://mlsanity.com/paper/2301.04246/) (2023)
- [Auditing large language models: a three-layered approach](https://mlsanity.com/paper/2302.08500/) (2023)
- [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://mlsanity.com/paper/2307.03718/) (2023)
- [Evaluating the Social Impact of Generative AI Systems in Systems and Society](https://mlsanity.com/paper/2306.05949/) (2023)
- [Eight Things to Know about Large Language Models](https://mlsanity.com/paper/2304.00612/) (2023)
- [Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance](https://mlsanity.com/paper/2206.04737/) (2022)
- [Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools](https://mlsanity.com/paper/2306.13952/) (2023)
- [Structured access: an emerging paradigm for safe AI deployment](https://mlsanity.com/paper/2201.05159/) (2022)
- [Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectives](https://mlsanity.com/paper/2311.09227/) (2023)
- [Democratising AI: Multiple Meanings, Goals, and Methods](https://mlsanity.com/paper/2303.12642/) (2023)
- [Why and How Governments Should Monitor AI Development](https://mlsanity.com/paper/2108.12427/) (2021)
- [Deployment Corrections: An incident response framework for frontier AI models](https://mlsanity.com/paper/2310.00328/) (2023)

## Built on by (most cited first, 11 of 11)

- [Black-Box Access is Insufficient for Rigorous AI Audits](https://mlsanity.com/paper/2401.14446/) (2024)
- [Logits of API-Protected LLMs Leak Proprietary Information](https://mlsanity.com/paper/2403.09539/) (2024)
- [Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities](https://mlsanity.com/paper/2502.05209/) (2025)
- [Navigating LLM Ethics: Advancements, Challenges, and Future Directions](https://mlsanity.com/paper/2406.18841/) (2024)
- [Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities](https://mlsanity.com/paper/2410.09114/) (2024)
- [Responsible Reporting for Frontier AI Development](https://mlsanity.com/paper/2404.02675/) (2024)
- [Societal Adaptation to Advanced AI](https://mlsanity.com/paper/2405.10295/) (2024)
- [Verification methods for international AI agreements](https://mlsanity.com/paper/2408.16074/) (2024)
- [The Necessity of AI Audit Standards Boards](https://mlsanity.com/paper/2404.13060/) (2024)
- [Towards Safe Multilingual Frontier AI](https://mlsanity.com/paper/2409.13708/) (2024)
- [Exposure is not manifestation: measurement target and output resolution jointly determine which behavioural-faithfulness evaluator wins](https://mlsanity.com/paper/2607.09306/) (2026)

## Abstract (arXiv)

With the increasing integration of frontier large language models (LLMs) into society and the economy, decisions related to their training, deployment, and use have far-reaching implications. These decisions should not be left solely in the hands of frontier LLM developers. LLM users, civil society and policymakers need trustworthy sources of information to steer such decisions for the better. Involving outside actors in the evaluation of these systems - what we term 'external scrutiny' - via red-teaming, auditing, and external researcher access, offers a solution. Though there are encouraging signs of increasing external scrutiny of frontier LLMs, its success is not assured. In this paper, we survey six requirements for effective external scrutiny of frontier AI systems and organize them under the ASPIRE framework: Access, Searching attitude, Proportionality to the risks, Independence, Resources, and Expertise. We then illustrate how external scrutiny might function throughout the AI lifecycle and offer recommendations to policymakers.