# Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Framework

arXiv: https://arxiv.org/abs/2605.10043 · HTML: https://mlsanity.com/paper/2605.10043/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [Positive-Unlabeled Learning with Non-Negative Risk Estimator](https://mlsanity.com/paper/1703.00593/) (2017)
- [Learning from positive and unlabeled data: a survey](https://mlsanity.com/paper/1811.04820/) (2018)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers](https://mlsanity.com/paper/2311.09180/) (2023)
- [Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts](https://mlsanity.com/paper/2406.10471/) (2024)
- [Binary Classifier Optimization for Large Language Model Alignment](https://mlsanity.com/paper/2404.04656/) (2024)
- [A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications](https://mlsanity.com/paper/2503.17003/) (2025)
- [Retrieval Augmented Generation with Collaborative Filtering for Personalized Text Generation](https://mlsanity.com/paper/2504.05731/) (2025)
- [Latent Inter-User Difference Modeling for LLM Personalization](https://mlsanity.com/paper/2507.20849/) (2025)
- [Multi-objective Large Language Model Alignment with Hierarchical Experts](https://mlsanity.com/paper/2505.20925/) (2025)
- [Personalized LLM Decoding via Contrasting Personal Preference](https://mlsanity.com/paper/2506.12109/) (2025)
- [E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning](https://mlsanity.com/paper/2604.09455/) (2026)
- [Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward](https://mlsanity.com/paper/2604.09748/) (2026)

## Built on by (most cited first, 1 of 1)

- [COPE: Continual Personalization of LLMs under Sparse User Feedback via User Embeddings and Self-Evaluation](https://mlsanity.com/paper/2609.26853/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) personalization aims to align model behaviors with individual user preferences. Existing methods often focus on isolated user histories, neglecting the essential role of inter-user differences. We propose C-BPO, a framework that personalizes LLMs via preference-calibrated binary signals. By treating target user data as positive feedback and other users' data as an auxiliary set of implicit negative signals, C-BPO captures distinct inter-user differences. To mitigate the preference overlap issue, where shared task knowledge is erroneously penalized, we derive an objective grounded in Positive-Unlabeled (PU) learning theory. This approach purifies negative signals by subtracting ``positive bias'', ensuring alignment with unique idiosyncrasies without compromising general helpfulness. Empirical experiments across various personalization tasks and backbone LLMs show C-BPO consistently outperforms baselines, demonstrating the efficacy of preference-calibrated binary signals in modeling inter-user differences.