# An Approach to Technical AGI Safety and Security

arXiv: https://arxiv.org/abs/2504.01849 · HTML: https://mlsanity.com/paper/2504.01849/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2025.

## Built on by (most cited first, 30 of 45)

- [Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety](https://mlsanity.com/paper/2507.11473/) (2025)
- [When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors](https://mlsanity.com/paper/2507.05246/) (2025)
- [CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring](https://mlsanity.com/paper/2505.23575/) (2025)
- [Building Production-Ready Probes For Gemini](https://mlsanity.com/paper/2601.11516/) (2026)
- [Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations](https://mlsanity.com/paper/2505.13763/) (2025)
- [Evaluating Frontier Models for Stealth and Situational Awareness](https://mlsanity.com/paper/2505.01420/) (2025)
- [Scaling Laws For Scalable Oversight](https://mlsanity.com/paper/2504.18530/) (2025)
- [Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorability](https://mlsanity.com/paper/2510.19851/) (2025)
- [Frontier AI's Impact on the Cybersecurity Landscape](https://mlsanity.com/paper/2504.05408/) (2025)
- [Virtual Agent Economies](https://mlsanity.com/paper/2509.10147/) (2025)
- [How to evaluate control measures for LLM agents? A trajectory from today to superintelligence](https://mlsanity.com/paper/2504.05259/) (2025)
- [Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Models](https://mlsanity.com/paper/2507.12428/) (2025)
- [Because we have LLMs, we Can and Should Pursue Agentic Interpretability](https://mlsanity.com/paper/2506.12152/) (2025)
- [Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges](https://mlsanity.com/paper/2507.19672/) (2025)
- [A Safety and Security Framework for Real-World Agentic Systems](https://mlsanity.com/paper/2511.21990/) (2025)
- [Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputs](https://mlsanity.com/paper/2506.14003/) (2025)
- [The Singapore Consensus on Global AI Safety Research Priorities](https://mlsanity.com/paper/2506.20702/) (2025)
- [Combining Cost-Constrained Runtime Monitors for AI Safety](https://mlsanity.com/paper/2507.15886/) (2025)
- [Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitors](https://mlsanity.com/paper/2512.11949/) (2025)
- [Semantic Structure in Large Language Model Embeddings](https://mlsanity.com/paper/2508.10003/) (2025)
- [Training Agents to Self-Report Misbehavior](https://mlsanity.com/paper/2602.22303/) (2026)
- [OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Models](https://mlsanity.com/paper/2505.21537/) (2025)
- [Embodied AI: Emerging Risks and Opportunities for Policy Action](https://mlsanity.com/paper/2509.00117/) (2025)
- [Improving Interactive In-Context Learning from Natural Language Feedback](https://mlsanity.com/paper/2602.16066/) (2026)
- [Human-AI Complementarity: A Goal for Amplified Oversight](https://mlsanity.com/paper/2510.26518/) (2025)
- [Intent Laundering: AI Safety Datasets Are Not What They Seem](https://mlsanity.com/paper/2602.16729/) (2026)
- [EMBGuard: Constructing Hazard-Aware Guardrails for Safe Planning in Embodied Agents](https://mlsanity.com/paper/2605.30924/) (2026)
- [Against racing to AGI: Cooperation, deterrence, and catastrophic risks](https://mlsanity.com/paper/2507.21839/) (2025)
- [From Firewalls to Frontiers: AI Red-Teaming is a Domain-Specific Evolution of Cyber Red-Teaming](https://mlsanity.com/paper/2509.11398/) (2025)
- [Psychological and behavioural responses in human-agent vs. human-human interactions: a systematic review and meta-analysis](https://mlsanity.com/paper/2509.21542/) (2025)

## Abstract (arXiv)

Artificial General Intelligence (AGI) promises transformative benefits but also presents significant risks. We develop an approach to address the risk of harms consequential enough to significantly harm humanity. We identify four areas of risk: misuse, misalignment, mistakes, and structural risks. Of these, we focus on technical approaches to misuse and misalignment. For misuse, our strategy aims to prevent threat actors from accessing dangerous capabilities, by proactively identifying dangerous capabilities, and implementing robust security, access restrictions, monitoring, and model safety mitigations. To address misalignment, we outline two lines of defense. First, model-level mitigations such as amplified oversight and robust training can help to build an aligned model. Second, system-level security measures such as monitoring and access control can mitigate harm even if the model is misaligned. Techniques from interpretability, uncertainty estimation, and safer design patterns can enhance the effectiveness of these mitigations. Finally, we briefly outline how these ingredients could be combined to produce safety cases for AGI systems.