mlsanityData to 7 Oct 2026

Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment

arXiv 2601.17329 ↗First posted 24 Jan 2026cs.LG cs.AIcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference-based alignment like Reinforcement Learning from Human Feedback (RLHF) learns from pairwise preferences, yet the labels are often noisy and inconsistent. Existing uncertainty-aware approaches weight preferences, but ignore a more fundamental factor: the reliability of the answers being compared. To address the problem, we propose Conformal Feedback Alignment (CFA), a framework that grounds preference weighting in the statistical guarantees of Conformal Prediction (CP). CFA quantifies answer-level reliability by constructing conformal prediction sets with controllable coverage and aggregates these reliabilities into principled weights for both DPO- and PPO-style training. Experiments across different datasets show that CFA improves alignment robustness and data efficiency, highlighting that modeling answer-side uncertainty complements preference-level weighting and yields more robust, data-efficient alignment. Codes are provided here.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 49 references that are in the map. First posted on arXiv 24 Jan 2026.

Builds on

31 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024WPO: Enhancing RLHF with Weighted Preference Optimization17 Jun
  2. 2024Uncertainty Aware Learning for Language Model Alignment7 Jun
  3. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  7. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  9. 2024API Is Enough: Conformal Prediction for Large Language Models Without Logit-Accesscited by 68
  10. 2024Linguistic Calibration of Long-Form Generationscited by 59
  11. 2022Understanding AdamW through Proximal Methods and Scale-Freenesscited by 46
  12. 2023Open-TI: Open Traffic Intelligence with Augmented Language Modelcited by 38
14 more
  1. 2023Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learningcited by 36
  2. 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 36
  3. 2024Prototypical Reward Network for Data-Efficient RLHFcited by 22
  4. 2024LLM Uncertainty Quantification through Directional Entailment Graph and Claim Level Response Augmentationcited by 22
  5. 2024CoMAL: Collaborative Multi-Agent Large Language Models for Mixed-Autonomy Trafficcited by 22
  6. 2024HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedbackcited by 15
  7. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  8. 2024Multi-objective Reinforcement learning from AI Feedbackcited by 9
  9. 2024Privacy-preserving Fine-tuning of Large Language Models through Flatnesscited by 6
  10. 2025MCQA-Eval: Efficient Confidence Evaluation in NLG with Gold-Standard Correctness Labelscited by 4
  11. 2025DeepShade: Enable Shade Simulation by Text-conditioned Image Generationcited by 4
  12. 2021The Online Pivot: Lessons Learned from Teaching a Text and Data Mining Course in Lockdown, Enhancing online Teaching with Pair Programming and Digital Badgescited by 2
  13. 2025cMALC-D: Contextual Multi-Agent LLM-Guided Curriculum Learning with Diversity-Based Context Blendingcited by 2
  14. 2025GE-Chat: A Graph Enhanced RAG Framework for Evidential Response Generation of LLMscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Every Response Counts: Quantifying Uncertainty of LLM-based Multi-Agent Systems through Tensor Decompositioncited by 5
  2. 2026Position: Uncertainty Quantification in LLMs is Just Unsupervised Clusteringcited by 0
  3. 2026STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hackingcited by 0
  4. 2026Range-GRPO: Policy Optimization via Pairwise Relations among Reward Intervalscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reliability-Aware LLM Alignment from Inconsistent Human Feedback
  2. 2025RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment
  3. 2023Preference Ranking Optimization for Human Alignment
  4. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  5. 2025Doubly Robust Alignment for Large Language Models
  6. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment
  7. 2025DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decomposition
  8. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment