# The Singapore Consensus on Global AI Safety Research Priorities

arXiv: https://arxiv.org/abs/2506.20702 · HTML: https://mlsanity.com/paper/2506.20702/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 93 of its 140 references that are in the map.
- First posted on arXiv 25 Jun 2025.

## Builds on (most cited first, 30 of 93)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [A Watermark for Large Language Models](https://mlsanity.com/paper/2301.10226/) (2023)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher](https://mlsanity.com/paper/2308.06463/) (2023)
- [Reasoning Models Don't Always Say What They Think](https://mlsanity.com/paper/2505.05410/) (2025)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [AI Alignment: A Comprehensive Survey](https://mlsanity.com/paper/2310.19852/) (2023)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [Trustworthy AI: From Principles to Practices](https://mlsanity.com/paper/2110.01167/) (2021)
- [Foundational Challenges in Assuring Alignment and Safety of Large Language Models](https://mlsanity.com/paper/2404.09932/) (2024)
- [Knowledge Editing for Large Language Models: A Survey](https://mlsanity.com/paper/2310.16218/) (2023)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [Open Problems in Mechanistic Interpretability](https://mlsanity.com/paper/2501.16496/) (2025)
- [Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models](https://mlsanity.com/paper/2406.10162/) (2024)
- [Model evaluation for extreme risks](https://mlsanity.com/paper/2305.15324/) (2023)
- [Multi-Agent Risks from Advanced AI](https://mlsanity.com/paper/2502.14143/) (2025)

## Built on by (most cited first, 10 of 10)

- [OpenFake: An Open Dataset and Platform Toward Real-World Deepfake Detection](https://mlsanity.com/paper/2509.09495/) (2025)
- [Toward a Safe Internet of Agents](https://mlsanity.com/paper/2512.00520/) (2025)
- [Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations](https://mlsanity.com/paper/2510.17256/) (2025)
- [Responsible AI Technical Report](https://mlsanity.com/paper/2509.20057/) (2025)
- [From Refusal to Recovery: A Control-Theoretic Approach to Generative AI Guardrails](https://mlsanity.com/paper/2510.13727/) (2025)
- [Comprehensive AI governance requires addressing non-model gains](https://mlsanity.com/paper/2606.00047/) (2026)
- [Failure Modes of Maximum Entropy RLHF](https://mlsanity.com/paper/2509.20265/) (2025)
- [Automated Analysis of Global AI Safety Initiatives: A Taxonomy-Driven LLM Approach](https://mlsanity.com/paper/2604.03533/) (2026)
- [An Instrument to Evaluate Governance Proposals: AI Policy Analysis at Scale](https://mlsanity.com/paper/2607.28094/) (2026)
- [Agency as a New Organizing Abstraction for Computer Science](https://mlsanity.com/paper/2610.17410/) (2026)

## Abstract (arXiv)

Rapidly improving AI capabilities and autonomy hold significant promise of transformation, but are also driving vigorous debate on how to ensure that AI is safe, i.e., trustworthy, reliable, and secure. Building a trusted ecosystem is therefore essential -- it helps people embrace AI with confidence and gives maximal space for innovation while avoiding backlash. The "2025 Singapore Conference on AI (SCAI): International Scientific Exchange on AI Safety" aimed to support research in this space by bringing together AI scientists across geographies to identify and synthesise research priorities in AI safety. This resulting report builds on the International AI Safety Report chaired by Yoshua Bengio and backed by 33 governments. By adopting a defence-in-depth model, this report organises AI safety research domains into three types: challenges with creating trustworthy AI systems (Development), challenges with evaluating their risks (Assessment), and challenges with monitoring and intervening after deployment (Control).