mlsanityData to 14 Oct 2026

Unsupervised Human Preference Learning

arXiv 2410.03731 ↗First posted 30 Sep 2024cs.CL cs.AIcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Large language models demonstrate impressive reasoning abilities but struggle to provide personalized content due to their lack of individual user preference information. Existing methods, such as in-context learning and parameter-efficient fine-tuning, fall short in capturing the complexity of human preferences, especially given the small, personal datasets individuals possess. In this paper, we propose a novel approach utilizing small parameter models as preference agents to generate natural language rules that guide a larger, pre-trained model, enabling efficient personalization. Our method involves a small, local "steering wheel" model that directs the outputs of a much larger foundation model, producing content tailored to an individual's preferences while leveraging the extensive knowledge and capabilities of the large model. Importantly, this personalization is achieved without the need to fine-tune the large model. Experimental results on email and article datasets, demonstrate that our technique significantly outperforms baseline personalization methods. By allowing foundation models to adapt to individual preferences in a data and compute-efficient manner, our approach paves the way for highly personalized language model applications.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 45 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

22 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  9. 2023LaMP: When Large Language Models Meet Personalizationcited by 437
  10. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  11. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  12. 2024Aligner: Efficient Alignment by Learning to Correctcited by 98
8 more
  1. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 95
  2. 2024Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuningcited by 90
  3. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 69
  4. 2017Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learningcited by 44
  5. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  6. 2024Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applicationscited by 25
  7. 2023Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Modelscited by 11
  8. 2024Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answeringcited by 4

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Aligning LLMs by Predicting Preferences from User Writing Samplescited by 11

Similar, not cited

Close in wording, no citation either way

  1. 2026Text as a Universal Interface for Transferable Personalization
  2. 2025POPI: Personalizing LLMs via Optimized Natural Language Preference Inference
  3. 2024Orchestrating LLMs with Different Personalizations
  4. 2024Personalized Language Modeling from Personalized Human Feedback
  5. 2026Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalization
  6. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment
  7. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
  8. 2026Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences