# Conformal Linguistic Calibration: Trading-off between Factuality and Specificity

arXiv: https://arxiv.org/abs/2502.19110 · HTML: https://mlsanity.com/paper/2502.19110/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 68 of its 90 references that are in the map.
- First posted on arXiv 26 Feb 2025.

## Builds on (most cited first, 30 of 68)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [Conformal Risk Control](https://mlsanity.com/paper/2208.02814/) (2022)
- [Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models](https://mlsanity.com/paper/2305.19187/) (2023)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [RARR: Researching and Revising What Language Models Say, Using Language Models](https://mlsanity.com/paper/2210.08726/) (2022)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://mlsanity.com/paper/2404.10774/) (2024)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [From r to Q^*: Your Language Model is Secretly a Q-Function](https://mlsanity.com/paper/2404.12358/) (2024)
- [Selective Question Answering under Domain Shift](https://mlsanity.com/paper/2006.09462/) (2020)
- [Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer](https://mlsanity.com/paper/1711.06664/) (2017)
- [Fine-tuning Language Models for Factuality](https://mlsanity.com/paper/2311.08401/) (2023)
- [Reducing conversational agents' overconfidence through linguistic calibration](https://mlsanity.com/paper/2012.14983/) (2020)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration](https://mlsanity.com/paper/2402.00367/) (2024)
- [Conformal Language Modeling](https://mlsanity.com/paper/2306.10193/) (2023)

## Built on by (most cited first, 20 of 20)

- [From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centered](https://mlsanity.com/paper/2506.07461/) (2025)
- [MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs](https://mlsanity.com/paper/2505.24858/) (2025)
- [Conformal Risk Training: End-to-End Optimization of Conformal Risk Control](https://mlsanity.com/paper/2510.08748/) (2025)
- [Conformal Policy Control](https://mlsanity.com/paper/2603.02196/) (2026)
- [Statistical Methods in Generative AI](https://mlsanity.com/paper/2509.07054/) (2025)
- [Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs](https://mlsanity.com/paper/2606.32032/) (2026)
- [A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluation](https://mlsanity.com/paper/2608.24419/) (2026)
- [Process Supervision of Confidence Margin for Calibrated LLM Reasoning](https://mlsanity.com/paper/2604.23333/) (2026)
- [Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?](https://mlsanity.com/paper/2605.28778/) (2026)
- [Credal Large Language Models for Semantic Commitment under Uncertainty](https://mlsanity.com/paper/2608.23244/) (2026)
- [Bonsai: Interpretable Tree-Adaptive Grounded Reasoning](https://mlsanity.com/paper/2504.03640/) (2025)
- [Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation](https://mlsanity.com/paper/2510.24870/) (2025)
- [Conformalized Large Language Models under Configuration Shift](https://mlsanity.com/paper/2608.01460/) (2026)
- [Sample, Align, Synthesize: Graph-Based Response Synthesis with ConGrs](https://mlsanity.com/paper/2510.03527/) (2025)
- [Disentangling Deception and Hallucination Failures in LLMs](https://mlsanity.com/paper/2602.14529/) (2026)
- [PROMPT2BOX:Improving LLM Weakness Discovery and Specificity Estimation by Uncovering Entailment Structure among Prompts](https://mlsanity.com/paper/2603.21438/) (2026)
- [Answer Only as Precisely as Justified: Calibrated Claim-Level Specificity Control for Agentic Systems](https://mlsanity.com/paper/2604.17487/) (2026)
- [Configurable Reward Model for Balanced Safety Alignment](https://mlsanity.com/paper/2605.30487/) (2026)
- [Occupancy-based Quantile Risk Control](https://mlsanity.com/paper/2609.03104/) (2026)
- [FrankenReport: Early Exiting in Long-Form Generation Using Expected Value of Computation](https://mlsanity.com/paper/2609.06320/) (2026)

## Abstract (arXiv)

Language model outputs are not always reliable, thus prompting research into how to adapt model responses based on uncertainty. Common approaches include: abstention, where models refrain from generating responses when uncertain; and linguistic calibration, where models hedge their statements using uncertainty quantifiers. However, abstention can withhold valuable information, while linguistically calibrated responses are often challenging to leverage in downstream tasks. We propose a unified view, Conformal Linguistic Calibration (CLC), which reinterprets linguistic calibration as answer set prediction. First we present a framework connecting abstention and linguistic calibration through the lens of linguistic pragmatics. We then describe an implementation of CLC that allows for controlling the level of imprecision in model responses. Results demonstrate our method produces calibrated outputs with conformal guarantees on factual accuracy. Further, our approach enables fine-tuning models to perform uncertainty-aware adaptive claim rewriting, offering a controllable balance between factuality and specificity.