# CLIPer: Tailoring Diverse User Preference via Classifier-Guided Inference-Time Personalization

arXiv: https://arxiv.org/abs/2605.07162 · HTML: https://mlsanity.com/paper/2605.07162/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 8 May 2026.

## Builds on (most cited first, 14 of 14)

- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [UltraFeedback: Boosting Language Models with Scaled AI Feedback](https://mlsanity.com/paper/2310.01377/) (2023)
- [Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging](https://mlsanity.com/paper/2310.11564/) (2023)
- [Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning](https://mlsanity.com/paper/2408.10075/) (2024)
- [Personalized Language Modeling from Personalized Human Feedback](https://mlsanity.com/paper/2402.05133/) (2024)
- [Decoding-Time Language Model Alignment with Multiple Objectives](https://mlsanity.com/paper/2406.18853/) (2024)
- [HYDRA: Model Factorization Framework for Black-Box LLM Personalization](https://mlsanity.com/paper/2406.02888/) (2024)
- [PersonalLLM: Tailoring LLMs to Individual Preferences](https://mlsanity.com/paper/2409.20296/) (2024)
- [PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences](https://mlsanity.com/paper/2406.08469/) (2024)
- [Transfer Q Star: Principled Decoding for LLM Alignment](https://mlsanity.com/paper/2405.20495/) (2024)
- [MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Models](https://mlsanity.com/paper/2403.17141/) (2024)
- [Classifiers are Better Experts for Controllable Text Generation](https://mlsanity.com/paper/2205.07276/) (2022)

## Abstract (arXiv)

Personalized LLMs can significantly enhance user experiences by tailoring responses to preferences such as helpfulness, conciseness, and humor. However, fine-tuning models to address all possible combinations of user preferences is computationally expensive and impractical. In this paper, we introduce CLIPer(Classifier-guided Inference-time Personalization), a lightweight personalization approach that leverages a classifier model to steer LLM generation dynamically to different user preferences at inference time. Our method eliminates the need for extensive fine-tuning, inducing negligible additional computational overhead while enabling more controllable and nuanced personalization across single and multi-dimensional preferences. Comprehensive empirical analyses demonstrate the scalability and effectiveness of our approach in delivering personalized language generation.