mlsanityData to 14 Oct 2026

Conformal Linguistic Calibration: Trading-off between Factuality and Specificity

arXiv 2502.19110 ↗First posted 26 Feb 2025cs.CL cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

Language model outputs are not always reliable, thus prompting research into how to adapt model responses based on uncertainty. Common approaches include: abstention, where models refrain from generating responses when uncertain; and linguistic calibration, where models hedge their statements using uncertainty quantifiers. However, abstention can withhold valuable information, while linguistically calibrated responses are often challenging to leverage in downstream tasks. We propose a unified view, Conformal Linguistic Calibration (CLC), which reinterprets linguistic calibration as answer set prediction. First we present a framework connecting abstention and linguistic calibration through the lens of linguistic pragmatics. We then describe an implementation of CLC that allows for controlling the level of imprecision in model responses. Results demonstrate our method produces calibrated outputs with conformal guarantees on factual accuracy. Further, our approach enables fine-tuning models to perform uncertainty-aware adaptive claim rewriting, offering a controllable balance between factuality and specificity.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 68 of its 90 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

68 of 90 references.

References · 68

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2019Language Models as Knowledge Bases?cited by 1,936
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  9. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  10. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  11. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  12. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
38 more
  1. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  2. 2024Measuring short-form factuality in large language modelscited by 373
  3. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  4. 2022Conformal Risk Controlcited by 299
  5. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  6. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 288
  7. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  8. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
  9. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
  10. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  11. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  12. 2020Selective Question Answering under Domain Shiftcited by 238
  13. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
  14. 2023Fine-tuning Language Models for Factualitycited by 221
  15. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  16. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  17. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  18. 2023Conformal Language Modelingcited by 170
  19. 2024Long-form factuality in large language modelscited by 155
  20. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 139
  21. 2024Language Models with Conformal Factuality Guaranteescited by 131
  22. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 117
  23. 2024Large language model validity via enhanced conformal prediction methodscited by 92
  24. 2019Nested conformal prediction and quantile out-of-bag ensemble methodscited by 89
  25. 2023Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkerscited by 87
  26. 2021Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibrationcited by 79
  27. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  28. 2022Generating Literal and Implied Subquestions to Fact-check Complex Claimscited by 66
  29. 2024Linguistic Calibration of Long-Form Generationscited by 60
  30. 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 59
  31. 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 54
  32. 2024Can AI Assistants Know What They Don't Know?cited by 50
  33. 2021Can NLI Models Verify QA Systems' Predictions?cited by 46
  34. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 44
  35. 2023"According to ...": Prompting Language Models Improves Quoting from Pre-Training Datacited by 42
  36. 2024A Closer Look at Claim Decompositioncited by 41
  37. 2019Quizbowl: The Case for Incremental Question Answeringcited by 36
  38. 2023Learning to Reject with a Fixed Predictor: Application to Decontextualizationcited by 32

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 13.

20 papers.

Sort:

By area:

  1. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  2. 2025MetaFaith: Faithful Natural Language Uncertainty Expression in LLMscited by 16
  3. 2025Conformal Risk Training: End-to-End Optimization of Conformal Risk Controlcited by 11
  4. 2026Conformal Policy Controlcited by 6
  5. 2025Statistical Methods in Generative AIcited by 4
  6. 2026Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMscited by 4
  7. 2026A Judge Should Know What Changed:Construct Validity for LLM-as-a-Judge Evaluationcited by 4
  8. 2026Process Supervision of Confidence Margin for Calibrated LLM Reasoningcited by 2
  9. 2026Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?cited by 2
  10. 2026Credal Large Language Models for Semantic Commitment under Uncertaintycited by 2
  11. 2025Bonsai: Interpretable Tree-Adaptive Grounded Reasoningcited by 1
  12. 2025Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generationcited by 1
8 more
  1. 2026Conformalized Large Language Models under Configuration Shiftcited by 1
  2. 2025Sample, Align, Synthesize: Graph-Based Response Synthesis with ConGrscited by 0
  3. 2026Disentangling Deception and Hallucination Failures in LLMscited by 0
  4. 2026PROMPT2BOX:Improving LLM Weakness Discovery and Specificity Estimation by Uncovering Entailment Structure among Promptscited by 0
  5. 2026Answer Only as Precisely as Justified: Calibrated Claim-Level Specificity Control for Agentic Systemscited by 0
  6. 2026Configurable Reward Model for Balanced Safety Alignmentcited by 0
  7. 2026Occupancy-based Quantile Risk Controlcited by 0
  8. 2026FrankenReport: Early Exiting in Long-Form Generation Using Expected Value of Computationcited by 0