# When Policies Change Probabilities: Modular Decision-Making for LLM Code Review

arXiv: https://arxiv.org/abs/2608.02677 · HTML: https://mlsanity.com/paper/2608.02677/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 2 Aug 2026.

## Builds on (most cited first, 10 of 10)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Calibrating Large Language Models with Sample Consistency](https://mlsanity.com/paper/2402.13904/) (2024)
- [Mapping from Meaning: Addressing the Miscalibration of Prompt-Sensitive Language Models](https://mlsanity.com/paper/2510.17028/) (2025)
- [When Agents Say One Thing and Do Another: Validating Elicited Beliefs from LLMs](https://mlsanity.com/paper/2602.06286/) (2026)
- [Are LLM Decisions Faithful to Verbal Confidence?](https://mlsanity.com/paper/2601.07767/) (2026)
- [Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges](https://mlsanity.com/paper/2605.06161/) (2026)
- [Same Answer, Different Confidence: Protocol Sensitivity in LLM Confidence Calibration](https://mlsanity.com/paper/2605.27752/) (2026)
- [SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review](https://mlsanity.com/paper/2607.06065/) (2026)

## Built on by (most cited first, 1 of 1)

- [Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces](https://mlsanity.com/paper/2609.37470/) (2026)

## Abstract (arXiv)

LLM code reviewers often estimate patch risk and make approval decisions in one prompt. A probability should depend on evidence; costs should determine the action taken from it. We test whether four deployed reviewer interfaces preserve this separation using 15,792 responses on 720 candidate patches, with one that passed and one that failed an archived test harness for each of 360 repository issues. In matched calls with the patch and monitor evidence fixed, replacing an equal-cost policy with a 10:1 false-accept policy changes reported failure probabilities by 13.6 to 16.9 percentage points on average. For every reviewer, the actions returned under the high-cost prompt are worse than rejecting all patches. Applying the same high-cost rule to probabilities elicited under equal costs reduces loss for all four systems, showing that probability elicitation itself contributes to the excess loss. We also evaluate a modular pipeline that elicits risk without policy information, combines an independent monitor score, and applies costs in code. Relative to calibrated reviewer-only scores, the pipeline improves average probability accuracy and, at equal costs, reduces mean loss by .073 per issue while accepting 58 to 68% of patches. At 10:1, it accepts none and matches reject-all. Downstream policy can therefore change the probability it is meant to use, motivating separate evaluation of risk, outside evidence, and action.