mlsanityData to 7 Oct 2026

Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuning

arXiv 2407.15762 ↗First posted 22 Jul 2024cs.LG cs.AI cs.CLcited by 42

Not on the map. Why?

Cited by 26 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reward-based finetuning is crucial for aligning language policies with intended behaviors (e.g., creativity and safety). A key challenge is to develop steerable language models that trade-off multiple (conflicting) objectives in a flexible and efficient manner. This paper presents Conditional Language Policy (CLP), a general framework for finetuning language models on multiple objectives. Building on techniques from multi-task training and parameter-efficient finetuning, CLP learn steerable models that effectively trade-off conflicting objectives at inference time. Notably, this does not require training or maintaining multiple models to achieve different trade-offs between the objectives. Through extensive experiments and ablations on two summarization datasets, we show that CLP learns steerable language models that outperform and Pareto-dominate the existing approaches for multi-objective finetuning.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

36 of 51 references.

References · 36

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
24 more
  1. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  2. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  3. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  4. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  5. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  6. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
  7. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  8. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 229
  9. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  10. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  11. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  12. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  13. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  14. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
  15. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  16. 2021Learning Neural Network Subspacescited by 82
  17. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  18. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  19. 2024Decoding-time Realignment of Language Modelscited by 67
  20. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
  21. 2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29
  22. 2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferencescited by 25
  23. 2019AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learningcited by 22
  24. 2021Learning a subspace of policies for online adaptation in Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 21; 2026: 13.

42 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  3. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 28
  4. 2025Language Model Personalization via Reward Factorizationcited by 27
  5. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
  6. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  7. 2025SMART: Simulated Students Aligned with Item Response Theory for Question Difficulty Predictioncited by 21
  8. 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 19
  9. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  10. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 14
  11. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  12. 2025Rethinking Diverse Human Preference Learning through Principal Component Analysiscited by 10
30 more
  1. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 10
  2. 2026Reward-free Alignment for Conflicting Objectivescited by 10
  3. 2024Diversity-Rewarded CFG Distillationcited by 9
  4. 2024The Central Role of the Loss Function in Reinforcement Learningcited by 8
  5. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 8
  6. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  7. 2025Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Modelscited by 6
  8. 2024COS-DPO: Conditioned One-Shot Multi-Objective Fine-Tuning Frameworkcited by 5
  9. 2024Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Bothcited by 5
  10. 2025Learning Steerable Clarification Policies with Collaborative Self-playcited by 5
  11. 2024Multi-objective Reinforcement Learning: A Tool for Pluralistic Alignmentcited by 4
  12. 2024Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processescited by 4
  13. 2025Mol-MoE: Training Preference-Guided Routers for Molecule Generationcited by 3
  14. 2026UniARM: Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignmentcited by 3
  15. 2026Property-driven Protein Inverse Folding With Multi-Objective Preference Alignmentcited by 3
  16. 2025MOSLIM:Align with diverse preferences in prompts through reward classificationcited by 2
  17. 2026Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuningcited by 2
  18. 2026Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimizationcited by 2
  19. 2025Steerable Chatbots: Exploring Personalization Control Interfaces via LLM Activation Steeringcited by 1
  20. 2025Simple Optimizers for Convex Aligned Multi-Objective Optimizationcited by 1
  21. 2025Multi-dimensional Preference Alignment by Conditioning Reward Itselfcited by 1
  22. 2026Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansioncited by 1
  23. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
  24. 2026Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuningcited by 0
  25. 2026Many Preferences, Few Policies: Towards Scalable Language Model Personalizationcited by 0
  26. 2026One Model for All: Multi-Objective Controllable Language Modelscited by 0
  27. 2026Reward Weighted Classifier-Free Guidance as Policy Improvement in Autoregressive Modelscited by 0
  28. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0
  29. 2026Risk-Conditioned Fine-Tuning of Large Language Modelscited by 0
  30. 2026The Missing Coefficients: Bayesian Pairwise Merging for Model Personalizationcited by 0