mlsanityData to 14 Oct 2026

Conformal Alignment: Knowing When to Trust Foundation Models with Guarantees

arXiv 2405.10301 ↗First posted 16 May 2024stat.ML cs.AI cs.LGcited by 54

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Before deploying outputs from foundation models in high-stakes tasks, it is imperative to ensure that they align with human values. For instance, in radiology report generation, reports generated by a vision-language model must align with human evaluations before their use in medical decision-making. This paper presents Conformal Alignment, a general framework for identifying units whose outputs meet a user-specified alignment criterion. It is guaranteed that on average, a prescribed fraction of selected units indeed meet the alignment criterion, regardless of the foundation model or the data distribution. Given any pre-trained model and new units with model-generated outputs, Conformal Alignment leverages a set of reference data with ground-truth alignment status to train an alignment predictor. It then selects new units whose predicted alignment scores surpass a data-dependent threshold, certifying their corresponding outputs as trustworthy. Through applications to question answering and radiology report generation, we demonstrate that our method is able to accurately identify units with trustworthy outputs via lightweight training over a moderate amount of reference data. En route, we investigate the informativeness of various features in alignment prediction and combine them with standard models to construct the alignment predictor.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 59 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

42 of 59 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Unintended Impacts of LLM Alignment on Global Representation22 Feb
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 40

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  9. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  10. 2021Ethical and social risks of harm from Language Modelscited by 966
  11. 2017Selective Classification for Deep Neural Networkscited by 852
  12. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
28 more
  1. 2018CoQA: A Conversational Question Answering Challengecited by 683
  2. 2016Distribution-Free Predictive Inference For Regressioncited by 668
  3. 2019Conformalized Quantile Regressioncited by 629
  4. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  5. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 337
  6. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  7. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  8. 2020Consistent Estimators for Learning to Defer to an Expertcited by 271
  9. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
  10. 2020Selective Question Answering under Domain Shiftcited by 238
  11. 2023Conformal Language Modelingcited by 170
  12. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 139
  13. 2024Language Models with Conformal Factuality Guaranteescited by 131
  14. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 126
  15. 2022Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completioncited by 102
  16. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 89
  17. 2023Uncertainty Quantification over Graph with Conformalized Graph Neural Networkscited by 79
  18. 2022Selection by Prediction with Conformal p-valuescited by 69
  19. 2024API Is Enough: Conformal Prediction for Large Language Models Without Logit-Accesscited by 69
  20. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  21. 2023Adaptation with Self-Evaluation to Improve Selective Prediction in LLMscited by 50
  22. 2024Uncertainty in Language Models: Assessment through Rank-Calibrationcited by 41
  23. 2023Model-free selective inference under covariate shift via weighted conformal p-valuescited by 29
  24. 2023TRAQ: Trustworthy Retrieval Augmented Question Answering via Conformal Predictioncited by 25
  25. 2024Non-Exchangeable Conformal Language Generation with Nearest Neighborscited by 21
  26. 2022Selective Prediction via Training Dynamicscited by 20
  27. 2023Uncertainty-aware Language Modeling for Selective Question Answeringcited by 16
  28. 2024Confidence on the Focal: Conformal Prediction with Selection-Conditional Coveragecited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 22; 2026: 20.

54 papers.

Sort:

By area:

  1. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
  2. 2024Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Modelscited by 57
  3. 2024Conformal Prediction: A Data Perspectivecited by 25
  4. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  5. 2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 16
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  7. 2024Optimized Conformal Selection: Powerful Selective Inference After Conformity Score Optimizationcited by 14
  8. 2026SCOPE: Selective Conformal Optimized Pairwise LLM Judgingcited by 14
  9. 2024MetaToken: Detecting Hallucination in Image Descriptions by Meta Classificationcited by 13
  10. 2024Are foundation models for computer vision good conformal predictors?cited by 13
  11. 2026Conformal Selective Prediction with General Risk Controlcited by 12
  12. 2024Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filteringcited by 10
38 more
  1. 2025Multivariate Conformal Selectioncited by 8
  2. 2025LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systemscited by 7
  3. 2024Adapting Prediction Sets to Distribution Shifts Without Labelscited by 6
  4. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 6
  5. 2025Conformal Prediction Beyond the Seen: A Missing Mass Perspective for Uncertainty Quantification in Generative Modelscited by 6
  6. 2024Semantic Consistency-Based Uncertainty Quantification for Factuality in Radiology Report Generationcited by 5
  7. 2025Exploring the Noise Robustness of Online Conformal Predictioncited by 5
  8. 2025Do Large Language Models (Really) Need Statistical Foundations?cited by 5
  9. 2025MCQA-Eval: Efficient Confidence Evaluation in NLG with Gold-Standard Correctness Labelscited by 4
  10. 2025Statistical Methods in Generative AIcited by 4
  11. 2025Multi-Condition Conformal Selectioncited by 4
  12. 2026BalanceRAG: Joint Risk Calibration for Cascaded Retrieval-Augmented Generationcited by 4
  13. 2026When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generationcited by 4
  14. 2025Calibrated Predictive Lower Bounds on Time-to-Unsafe-Sampling in LLMscited by 3
  15. 2025Filtering with Confidence: When Data Augmentation Meets Conformal Predictioncited by 3
  16. 2025Model-agnostic Selective Labeling with Provable Statistical Guaranteescited by 3
  17. 2024Streamlining Conformal Information Retrieval via Score Refinementcited by 2
  18. 2025High Accuracy, Less Talk (HALT): Reliable LLMs through Capability-Aligned Finetuningcited by 2
  19. 2025Non-Asymptotic Analysis of Efficiency in Conformalized Regressioncited by 2
  20. 2025Distribution-informed Online Conformal Predictioncited by 2
  21. 2025Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospectscited by 1
  22. 2025SAFER: A Calibrated Risk-Aware Multimodal Recommendation Model for Dynamic Treatment Regimescited by 1
  23. 2025Reliable Inference in Edge-Cloud Model Cascades via Conformal Alignmentcited by 1
  24. 2026Margin-Adaptive Confidence Ranking for Reliable LLM Judgementcited by 1
  25. 2026Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guaranteescited by 1
  26. 2026Localize-Then-Decide Guarantees for LLM Judgmentscited by 1
  27. 2025Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Controlcited by 0
  28. 2025Smoothing-Based Conformal Prediction for Balancing Efficiency and Interpretabilitycited by 0
  29. 2025Ensuring Calibration Robustness in Split Conformal Prediction Under Adversarial Attackscited by 0
  30. 2026AviationLMM: A Large Multimodal Foundation Model for Civil Aviationcited by 0
  31. 2026Online Selective Conformal Prediction with Asymmetric Rules: A Permutation Test Approachcited by 0
  32. 2026Generative Score Inference for Multimodal Datacited by 0
  33. 2026Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibrationcited by 0
  34. 2026Position: Uncertainty Quantification in LLMs is Just Unsupervised Clusteringcited by 0
  35. 2026Provable Joint Decontamination for Benchmarking Multiple Large Language Modelscited by 0
  36. 2026Online Safety Monitoring for LLMscited by 0
  37. 2026Asymptotic Risk Calibration for Selective Question Answeringcited by 0
  38. 2026From Uncertainty to Clinical Risk: Severity-Aware Conformal Planning for Interactive Medical Diagnosiscited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Graph-Supervised Hierarchical Clinical Alignment for Radiology Report Generation with Large Language Models
  2. 2026LEAD: Layer-wise Expert-aligned Decoding for Faithful Radiology Report Generation
  3. 2025S2D-ALIGN: Shallow-to-Deep Auxiliary Learning for Anatomically-Grounded Radiology Report Generation
  4. 2026Alignment Forecasting: Predicting Misalignment From Training Data
  5. 2024Aligner: Efficient Alignment by Learning to Correct
  6. 2026Learning to See Locally and Align Clinically with Pathology Semantics for Radiology Report Generation
  7. 2025Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generation
  8. 2026LLM-Guided Diagnostic Evidence Alignment for Medical Vision-Language Pretraining under Limited Pairing