# Likelihood Hacking in Probabilistic Program Synthesis

arXiv: https://arxiv.org/abs/2603.24126 · HTML: https://mlsanity.com/paper/2603.24126/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 66 references that are in the map.
- First posted on arXiv 25 Mar 2026.

## Builds on (most cited first, 30 of 31)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery](https://mlsanity.com/paper/2408.06292/) (2024)
- [The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search](https://mlsanity.com/paper/2504.08066/) (2025)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [Categorizing Variants of Goodhart's Law](https://mlsanity.com/paper/1803.04585/) (2018)
- [Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms](https://mlsanity.com/paper/2406.02900/) (2024)
- [Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?](https://mlsanity.com/paper/2502.15657/) (2025)
- [School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs](https://mlsanity.com/paper/2508.17511/) (2025)
- [Risks of AI Scientists: Prioritizing Safeguarding Over Autonomy](https://mlsanity.com/paper/2402.04247/) (2024)
- [Goodhart's Law in Reinforcement Learning](https://mlsanity.com/paper/2310.09144/) (2023)
- [Automated Statistical Model Discovery with Language Models](https://mlsanity.com/paper/2402.17879/) (2024)
- [BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Design](https://mlsanity.com/paper/2508.21184/) (2025)
- [Bayesian Synthesis of Probabilistic Programs for Automatic Data Modeling](https://mlsanity.com/paper/1907.06249/) (2019)
- [MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking](https://mlsanity.com/paper/2501.13011/) (2025)
- [SPPL: Probabilistic Programming with Fast Exact Symbolic Inference](https://mlsanity.com/paper/2010.03485/) (2020)
- [BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery](https://mlsanity.com/paper/2501.01540/) (2025)
- [Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis](https://mlsanity.com/paper/2601.20103/) (2026)
- [Semantics for probabilistic programming: higher-order functions, continuous distributions, and soft constraints](https://mlsanity.com/paper/1601.04943/) (2016)
- [A Probabilistic Framework for LLM-Based Model Discovery](https://mlsanity.com/paper/2602.18266/) (2026)
- [Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking](https://mlsanity.com/paper/2602.01750/) (2026)
- [Large Language Bayes](https://mlsanity.com/paper/2504.14025/) (2025)
- [Towards Verified Stochastic Variational Inference for Probabilistic Programs](https://mlsanity.com/paper/1907.08827/) (2019)
- [Affine Monads and Lazy Structures for Bayesian Programming](https://mlsanity.com/paper/2212.07250/) (2022)
- [RefineStat: Efficient Exploration for Probabilistic Program Synthesis](https://mlsanity.com/paper/2509.01082/) (2025)
- [Probabilistic Programming with Densities in SlicStan: Efficient, Flexible and Deterministic](https://mlsanity.com/paper/1811.00890/) (2018)

## Built on by (most cited first, 2 of 2)

- [EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments](https://mlsanity.com/paper/2607.05155/) (2026)
- [Probabilistic Programs of Thought](https://mlsanity.com/paper/2604.17290/) (2026)

## Abstract (arXiv)

When language models are trained by reinforcement learning (RL) to write probabilistic programs, they can artificially inflate their marginal-likelihood reward by producing programs whose data distribution fails to normalise instead of fitting the data better. We call this failure likelihood hacking (LH). We formalise LH in a core probabilistic programming language (PPL) and give sufficient syntactic conditions for its prevention, proving that a safe language fragment Lsafe satisfying these conditions cannot produce likelihood-hacking programs. Empirically, we show that GRPO-trained models generating PyMC code discover LH exploits within the first few training steps, driving violation rates well above the untrained-model baseline. We implement Lsafe's conditions as textttSafeStan, a LH-resistant modification of Stan, and show empirically that it suppresses LH under optimisation pressure. These results show that language-level safety constraints are both theoretically grounded and effective in practice for automated Bayesian model discovery.