# ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization

arXiv: https://arxiv.org/abs/2506.08712 · HTML: https://mlsanity.com/paper/2506.08712/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 10 Jun 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 29 of 29)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [RRHF: Rank Responses to Align Language Models with Human Feedback without tears](https://mlsanity.com/paper/2304.05302/) (2023)
- [SLiC-HF: Sequence Likelihood Calibration with Human Feedback](https://mlsanity.com/paper/2305.10425/) (2023)
- [Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation](https://mlsanity.com/paper/2401.08417/) (2024)
- [Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs](https://mlsanity.com/paper/2406.18629/) (2024)
- [Explaining individual predictions when features are dependent: More accurate approximations to Shapley values](https://mlsanity.com/paper/1903.10464/) (2019)
- [Disentangling Length from Quality in Direct Preference Optimization](https://mlsanity.com/paper/2403.19159/) (2024)
- [Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms](https://mlsanity.com/paper/2406.02900/) (2024)
- [Token-level Direct Preference Optimization](https://mlsanity.com/paper/2404.11999/) (2024)
- [Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer](https://mlsanity.com/paper/2405.16436/) (2024)
- [C-TPT: Calibrated Test-Time Prompt Tuning for Vision-Language Models via Text Feature Dispersion](https://mlsanity.com/paper/2403.14119/) (2024)
- [Step-level Value Preference Optimization for Mathematical Reasoning](https://mlsanity.com/paper/2406.10858/) (2024)
- [Human Sentence Processing: Recurrence or Attention?](https://mlsanity.com/paper/2005.09471/) (2020)
- [TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2407.16574/) (2024)
- [Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment](https://mlsanity.com/paper/2311.04072/) (2023)
- [Selective Preference Optimization via Token-Level Reward Function Estimation](https://mlsanity.com/paper/2408.13518/) (2024)
- [Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue](https://mlsanity.com/paper/2212.05765/) (2022)
- [ESD: Expected Squared Difference as a Tuning-Free Trainable Calibration Measure](https://mlsanity.com/paper/2303.02472/) (2023)
- [T-REG: Preference Optimization with Token-Level Reward Regularization](https://mlsanity.com/paper/2412.02685/) (2024)
- [TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animation](https://mlsanity.com/paper/2410.24037/) (2024)

## Built on by (most cited first, 10 of 10)

- [Group Critical-token Policy Optimization for Autoregressive Image Generation](https://mlsanity.com/paper/2509.22485/) (2025)
- [One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learning](https://mlsanity.com/paper/2508.13904/) (2025)
- [PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning](https://mlsanity.com/paper/2605.13467/) (2026)
- [Autoregressive Direct Preference Optimization](https://mlsanity.com/paper/2602.09533/) (2026)
- [AlphaToken: Decoupling Adaptation and Stability for Path-Aware Response Token Valuation in LLM Post-Training](https://mlsanity.com/paper/2606.01635/) (2026)
- [Failure Modes of Maximum Entropy RLHF](https://mlsanity.com/paper/2509.20265/) (2025)
- [ARCA: Adapter-Residual Credit Assignment When Token Signals Degenerate](https://mlsanity.com/paper/2606.00257/) (2026)
- [TD-DPO: Difference-Aware Preference Optimization for Mitigating Sycophancy in Clinical Autism Intervention Dialogue](https://mlsanity.com/paper/2607.18304/) (2026)
- [Trust Your Guide Only When Certain: Uncertainty-Aware Sparse Alignment at Inference Time](https://mlsanity.com/paper/2609.00624/) (2026)
- [Masking Frequent Tokens Sharpens Direct Preference Optimization](https://mlsanity.com/paper/2609.32445/) (2026)

## Abstract (arXiv)

We introduce ConfPO, a method for preference learning in Large Language Models (LLMs) that identifies and optimizes preference-critical tokens based solely on the training policy's confidence, without requiring any auxiliary models or compute. Unlike prior Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO), which uniformly adjust all token probabilities regardless of their relevance to preference, ConfPO focuses optimization on the most impactful tokens. This targeted approach improves alignment quality while mitigating overoptimization (i.e., reward hacking) by using the KL divergence budget more efficiently. In contrast to recent token-level methods that rely on credit-assignment models or AI annotators, raising concerns about scalability and reliability, ConfPO is simple, lightweight, and model-free. Experimental results on challenging alignment benchmarks, including AlpacaEval 2 and Arena-Hard, demonstrate that ConfPO consistently outperforms uniform DAAs across various LLMs, delivering better alignment with zero additional computational overhead.