mlsanityData to 7 Oct 2026

An Approach to Technical AGI Safety and Security

arXiv 2504.01849 ↗First posted 2 Apr 2025cs.AI cs.CY cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Artificial General Intelligence (AGI) promises transformative benefits but also presents significant risks. We develop an approach to address the risk of harms consequential enough to significantly harm humanity. We identify four areas of risk: misuse, misalignment, mistakes, and structural risks. Of these, we focus on technical approaches to misuse and misalignment. For misuse, our strategy aims to prevent threat actors from accessing dangerous capabilities, by proactively identifying dangerous capabilities, and implementing robust security, access restrictions, monitoring, and model safety mitigations. To address misalignment, we outline two lines of defense. First, model-level mitigations such as amplified oversight and robust training can help to build an aligned model. Second, system-level security measures such as monitoring and access control can mitigate harm even if the model is misaligned. Techniques from interpretability, uncertainty estimation, and safer design patterns can enhance the effectiveness of these mitigations. Finally, we briefly outline how these ingredients could be combined to produce safety cases for AGI systems.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Apr 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 30; 2026: 12.

42 papers.

Sort:

By area:

  1. 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 222
  2. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorscited by 81
  3. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 49
  4. 2026Building Production-Ready Probes For Geminicited by 38
  5. 2025Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activationscited by 37
  6. 2025Evaluating Frontier Models for Stealth and Situational Awarenesscited by 30
  7. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  8. 2025How to evaluate control measures for LLM agents? A trajectory from today to superintelligencecited by 20
  9. 2025Scaling Laws For Scalable Oversightcited by 18
  10. 2025Can Reasoning Models Obfuscate Reasoning? Stress-Testing Chain-of-Thought Monitorabilitycited by 18
  11. 2025Virtual Agent Economiescited by 17
  12. 2025Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Modelscited by 16
30 more
  1. 2025Because we have LLMs, we Can and Should Pursue Agentic Interpretabilitycited by 14
  2. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  3. 2025A Safety and Security Framework for Real-World Agentic Systemscited by 11
  4. 2025Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputscited by 10
  5. 2025Combining Cost-Constrained Runtime Monitors for AI Safetycited by 10
  6. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 9
  7. 2025Semantic Structure in Large Language Model Embeddingscited by 7
  8. 2025Neural Chameleons: Language Models Can Learn to Hide Their Thoughts from Unseen Activation Monitorscited by 7
  9. 2025OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Modelscited by 5
  10. 2026Improving Interactive In-Context Learning from Natural Language Feedbackcited by 5
  11. 2025Embodied AI: Emerging Risks and Opportunities for Policy Actioncited by 3
  12. 2025Human-AI Complementarity: A Goal for Amplified Oversightcited by 3
  13. 2026Training Agents to Self-Report Misbehaviorcited by 3
  14. 2025Against racing to AGI: Cooperation, deterrence, and catastrophic riskscited by 2
  15. 2025From Firewalls to Frontiers: AI Red-Teaming is a Domain-Specific Evolution of Cyber Red-Teamingcited by 2
  16. 2025Psychological and behavioural responses in human-agent vs. human-human interactions: a systematic review and meta-analysiscited by 2
  17. 2026Position: Behavioural Assurance Cannot Verify the Safety Claims Governance Now Demandscited by 2
  18. 2025Contemplative Artificial Intelligencecited by 1
  19. 2025AI Alignment Strategies from a Risk Perspective: Independent Safety Mechanisms or Shared Failures?cited by 1
  20. 2025From Refusal to Recovery: A Control-Theoretic Approach to Generative AI Guardrailscited by 1
  21. 2026Intent Laundering: AI Safety Datasets Are Not What They Seemcited by 1
  22. 2026The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languagescited by 1
  23. 2026EMBGuard: Constructing Hazard-Aware Guardrails for Safe Planning in Embodied Agentscited by 1
  24. 2025Dimensional Characterization and Pathway Modeling for Catastrophic AI Riskscited by 0
  25. 2025AI Consciousness and Existential Riskcited by 0
  26. 2026Stress-Testing Alignment Audits With Prompt-Level Strategic Deceptioncited by 0
  27. 2026RCTs for Frontier AI Governance: Methodological Challenges and Solutions for Human Uplift Studiescited by 0
  28. 2026Questionnaire Responses Do not Capture the Safety of AI Agentscited by 0
  29. 2026The Possibility of Artificial Intelligence Becoming a Subject and the Alignment Problemcited by 0
  30. 2026Ambient Persuasion in a Deployed AI Agent: Unauthorized Escalation Following Routine Non-Adversarial Content Exposurecited by 0