mlsanityData to 14 Oct 2026

Conformal Tail Risk Control for Large Language Model Alignment

arXiv 2502.20285 ↗First posted 27 Feb 2025cs.LG stat.MLcited by 8

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Recent developments in large language models (LLMs) have led to their widespread usage for various tasks. The prevalence of LLMs in society implores the assurance on the reliability of their performance. In particular, risk-sensitive applications demand meticulous attention to unexpectedly poor outcomes, i.e., tail events, for instance, toxic answers, humiliating language, and offensive outputs. Due to the costly nature of acquiring human annotations, general-purpose scoring models have been created to automate the process of quantifying these tail events. This phenomenon introduces potential human-machine misalignment between the respective scoring mechanisms. In this work, we present a lightweight calibration framework for blackbox models that ensures the alignment of humans and machines with provable guarantees. Our framework provides a rigorous approach to controlling any distortion risk measure that is characterized by a weighted average of quantiles of the loss incurred by the LLM with high confidence. The theoretical foundation of our method relies on the connection between conformal risk control and a traditional family of statistics, i.e., L-statistics. To demonstrate the utility of our framework, we conduct comprehensive experiments that address the issue of human-machine misalignment.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 29 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

15 of 29 references.

References · 15

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  6. 2019Conformal Prediction Under Covariate Shiftcited by 519
  7. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 288
  8. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
  9. 2020Conformal Inference of Counterfactuals and Individual Treatment Effectscited by 151
  10. 2021Testing for Outliers with Conformal p-valuescited by 134
  11. 2024Language Models with Conformal Factuality Guaranteescited by 131
  12. 2024Large language model validity via enhanced conformal prediction methodscited by 92
3 more
  1. 2021Conformalized Survival Analysiscited by 53
  2. 2023Prompt Risk Control: A Rigorous Framework for Responsible Deployment of Large Language Modelscited by 17
  3. 2022Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictionscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Modelscited by 12
  2. 2025Conformal Risk Training: End-to-End Optimization of Conformal Risk Controlcited by 11
  3. 2026Adversarially Robust Control of Conditional Value-at-Risk via Rockafellar-Uryasev Conformal Inferencecited by 2
  4. 2026Calibrating Conservatism for Scalable Oversightcited by 0
  5. 2026Occupancy-based Quantile Risk Controlcited by 0
  6. 2026StepCOPS: Closed-Testing Lower-Tail Certificates for Language-Model Policy Selectioncited by 0
  7. 2026Prioritizing Repeated LLM Evaluation for Hidden Failure Discoverycited by 0
  8. 2026Quantifying Behavioral Tails in Black-Box Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  2. 2026DNAlign: Dynamic Null-Space Safe Alignment for LLMs
  3. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
  4. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  5. 2025Safety Alignment Depth in Large Language Models: A Markov Chain Perspective
  6. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching
  7. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysis
  8. 2025Investigating the Impact of Quantization Methods on the Safety and Reliability of Large Language Models