# Jailbreaks for Black-Box Uncertainty Quantification in Large Reasoning Models

arXiv: https://arxiv.org/abs/2609.35350 · HTML: https://mlsanity.com/paper/2609.35350/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 24 of its 66 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 24 of 24)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://mlsanity.com/paper/1806.08730/) (2018)
- [Jailbreak Attacks and Defenses Against Large Language Models: A Survey](https://mlsanity.com/paper/2407.04295/) (2024)
- [On Verbalized Confidence Scores for LLMs](https://mlsanity.com/paper/2412.14737/) (2024)
- [Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMs](https://mlsanity.com/paper/2505.04806/) (2025)
- [Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting](https://mlsanity.com/paper/2310.11732/) (2023)
- [Just rephrase it! Uncertainty estimation in closed-source language models via multiple rephrased queries](https://mlsanity.com/paper/2405.13907/) (2024)
- [Large Language Model Confidence Estimation via Black-Box Access](https://mlsanity.com/paper/2406.04370/) (2024)
- [Uncertainty is Fragile: Manipulating Uncertainty in Large Language Models](https://mlsanity.com/paper/2407.11282/) (2024)
- [The 2025 Foundation Model Transparency Index](https://mlsanity.com/paper/2512.10169/) (2025)
- [LLMs are Overconfident: Evaluating Confidence Interval Calibration with FermiEval](https://mlsanity.com/paper/2510.26995/) (2025)
- [Calibration Attacks: A Comprehensive Study of Adversarial Attacks on Model Confidence](https://mlsanity.com/paper/2401.02718/) (2024)
- [SeSE: Black-Box Uncertainty Quantification for Large Language Models Based on Structural Information Theory](https://mlsanity.com/paper/2511.16275/) (2025)
- [Safety Instincts: LLMs Learn to Trust Their Internal Compass for Self-Defense](https://mlsanity.com/paper/2510.01088/) (2025)
- [On the Robustness of Verbal Confidence of LLMs in Adversarial Attacks](https://mlsanity.com/paper/2507.06489/) (2025)

## Abstract (arXiv)

While Large Reasoning Models (LRMs) excel at complex reasoning, alignment through reinforcement learning often induces systemic overconfidence. In production environments, where logits may be unavailable, robust black-box uncertainty quantification (UQ) is essential for trustworthiness and safety. Focusing on question-answering for LRMs, we show that existing black-box methods, such as paraphrase-based self-consistency and confidence verbalization, offer little to no improvement over simple repeated sampling, suggesting that alignment suppresses useful output variability. We introduce prompt-level relaxation operators that broaden the model's effective output distribution by approximating the effect of an optimal policy obtained with a stronger KL-regularization parameter, hence closer to the reference model. Theoretically, we demonstrate that relaxation improves calibration. We propose Jailbreak for Uncertainty (J4U), a jailbreak-derived technique for UQ that empirically reproduces the behavioral signatures predicted by our relaxation theory. Across 3 datasets and 4 LRMs, including a closed-source production model, J4U's improvement over repeated sampling achieves statistical significance in up to 6 times more LRM-dataset-metric settings than the strongest black-box UQ state-of-the-art baseline we evaluate, with average ECE reductions up to 5 times larger. These results provide a practical tool for UQ in black-box LRM deployment.