mlsanityData to 14 Oct 2026

Can a Bayesian Oracle Prevent Harm from an Agent?

arXiv 2408.05284 ↗First posted 9 Aug 2024cs.AI cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Is there a way to design powerful AI systems based on machine learning methods that would satisfy probabilistic safety guarantees? With the long-term goal of obtaining a probabilistic guarantee that would apply in every context, we consider estimating a context-dependent bound on the probability of violating a given safety specification. Such a risk evaluation would need to be performed at run-time to provide a guardrail against dangerous actions of an AI. Noting that different plausible hypotheses about the world could produce very different outcomes, and because we do not know which one is right, we derive bounds on the safety violation probability predicted under the true but unknown hypothesis. Such bounds could be used to reject potentially dangerous actions. Our main results involve searching for cautious but plausible hypotheses, obtained by a maximization that involves Bayesian posteriors over hypotheses. We consider two forms of this result, in the i.i.d. case and in the non-i.i.d. case, and conclude with open problems towards turning such theoretical results into practical AI guardrails.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 31 of its 51 references that are in the map. First posted on arXiv 9 Aug 2024.

Builds on

31 of 51 references.

References · 31

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  5. 2022Bayesian Structure Learning with Generative Flow Networkscited by 153
  6. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 117
  7. 2023Amortizing intractable inference in large language modelscited by 97
  8. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 96
  9. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 90
  10. 2021Consequences of Misaligned AIcited by 80
  11. 2023Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programscited by 80
  12. 2024Amortizing intractable inference in diffusion models for vision, language, and controlcited by 72
19 more
  1. 2023Robust Scheduling with GFlowNetscited by 69
  2. 2022Safe Reinforcement Learning via Shielding under Partial Observabilitycited by 56
  3. 2024Ant Colony Sampling with GFlowNets for Combinatorial Optimizationcited by 53
  4. 2023Joint Bayesian Inference of Graphical Structure and Parameters with a Single Generative Flow Networkcited by 50
  5. 2023Local Search GFlowNetscited by 48
  6. 2024Improved off-policy training of diffusion samplerscited by 48
  7. 2023Training Chain-of-Thought via Latent-Variable Inferencecited by 47
  8. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  9. 2023Goodhart's Law in Reinforcement Learningcited by 41
  10. 2023GFlowNet-EM for learning compositional latent variable modelscited by 40
  11. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 37
  12. 2022Reward Gaming in Conditional Text Generationcited by 30
  13. 2022Counterfactual harmcited by 29
  14. 2023PhyloGFN: Phylogenetic inference with generative flow networkscited by 22
  15. 2023STARC: A General Framework For Quantifying Differences Between Reward Functionscited by 15
  16. 2020Pessimism About Unknown Unknowns Inspires Conservatismcited by 13
  17. 2022Quantifying Harmcited by 10
  18. 2024Latent Logic Tree Extraction for Event Sequence Explanation from LLMscited by 9
  19. 2021Fully General Online Imitation Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 4.

13 papers.

Sort:

By area:

  1. 2025General agents contain world modelscited by 17
  2. 2025Paradigms of AI Evaluation: Mapping Goals, Methodologies and Culturecited by 11
  3. 2025PredictaBoard: Benchmarking LLM Score Predictabilitycited by 7
  4. 2025Shh, don't say that! Domain Certification in LLMscited by 7
  5. 2026From AGI to ASIcited by 4
  6. 2025The Limits of Predicting Agents from Behaviourcited by 3
  7. 2026Position: agentic AI orchestration should be Bayes-consistentcited by 3
  8. 2025Free Energy Risk Metrics for Systemically Safe AI: Gatekeeping Multi-Agent Studycited by 2
  9. 2025The Strong, Weak and Benign Goodhart's law. An independence-free and paradigm-agnostic formalisationcited by 2
  10. 2025Can Safety Fine-Tuning Be More Principled? Lessons Learned from Cybersecuritycited by 1
  11. 2026Unbiased Canonical Set-Valued Oracles Via Lattice Theorycited by 1
  12. 2025Guided by Guardrails: Control Barrier Functions as Safety Instructors for Robotic Learningcited by 0
1 more
  1. 2026Sustaining AI safety: Control-theoretic external impossibility, intrinsic necessity, and structural requirementscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Probabilistic Shielding for Safe Reinforcement Learning
  2. 2026The Off-Support Barrier: Why Semantic Safety Constraints Are Not Learning-Problem Invariants, and What Follows for Prior Design, Containment, and Verification
  3. 2025Domain-Agnostic Scalable AI Safety Ensuring Framework
  4. 2026SafetyDrift: Predicting When AI Agents Cross the Line Before They Actually Do
  5. 2026Efficient and Sound Probabilistic Verification for AI Agents
  6. 2025ProSh: Probabilistic Shielding for Model-free Reinforcement Learning
  7. 2026Infra-Bayesian Reinforcement Learning Agents Outperform Classical RL For Worst-Case Robustness
  8. 2026Rules or Character? Scaling Laws for AI Safety Design