# ConformalGuard: False-Positive-Controlled Action Gating for LM Classifiers

arXiv: https://arxiv.org/abs/2610.14365 · HTML: https://mlsanity.com/paper/2610.14365/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 20 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 11 of 11)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2107.07511/) (2021)
- [Classification with Valid and Adaptive Coverage](https://mlsanity.com/paper/2006.02544/) (2020)
- [Conformal prediction beyond exchangeability](https://mlsanity.com/paper/2202.13415/) (2022)
- [Conformal Risk Control](https://mlsanity.com/paper/2208.02814/) (2022)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey](https://mlsanity.com/paper/2503.15850/) (2025)
- [ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guarantees](https://mlsanity.com/paper/2407.00499/) (2024)
- [A Survey of Uncertainty Estimation Methods on Large Language Models](https://mlsanity.com/paper/2503.00172/) (2025)
- [Conformal Prediction for Natural Language Processing: A Survey](https://mlsanity.com/paper/2405.01976/) (2024)
- [TECP: Token-Entropy Conformal Prediction for LLMs](https://mlsanity.com/paper/2509.00461/) (2025)

## Abstract (arXiv)

Language model classifiers are increasingly deployed for high-stakes decisions such as phishing screening, content moderation, and fraud triage, yet validation-set thresholds provide no finite-sample control of deployment-time false positives. We present ConformalGuard, a model-agnostic calibration layer that wraps a fixed scoring function with finite-sample control over the rate at which legitimate inputs trigger an intervention tier. Our contribution is empirical and operational, not a new conformal theorem: zero-shot LM API scores can be coarse enough (10--11 unique values here) to make temperature scaling and isotonic regression operationally identical to raw thresholding for low-FPR action gating. We compare these methods with conformal calibration for fine-tuned BERT, Claude Sonnet 4, and GPT-4o on public email corpora. For Claude, conformal calibration reduces FPR from 0.040 to 0.008 while recall falls from 0.938 to 0.875. For deterministic GPT-4o scores, the 1% operating point is too conservative to deploy (recall 0.564); three-run stochastic averaging raises recall to 0.732 at FPR 0.005. At 2% prevalence with BERT, conformal calibration removes 25 false-positive reviews per 10,000 emails (95% CI [10, 43]). The guarantee is legitimate-event-rate control under exchangeability; discrimination, workload effects, and robustness to shift remain empirical.