mlsanityData to 14 Oct 2026

Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback

arXiv 2603.12595 ↗First posted 13 Mar 2026cs.LG cs.AIcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is a widely used approach to align large-scale AI systems with human values. However, RLHF typically assumes a single, universal reward, which overlooks diverse preferences and limits personalization. Variational Preference Learning (VPL) seeks to address this by introducing user-specific latent variables. Despite its promise, we found that VPL suffers from posterior collapse. While this phenomenon is well known in VAEs, it has not previously been identified in preference learning frameworks. Under sparse preference data and with overly expressive decoders, VPL may cause latent variables to be ignored, reverting to a single-reward model. To overcome this limitation, we propose Swap-guided Preference Learning (SPL). The key idea is to construct fictitious swap annotators and use the mirroring property of their preferences to guide the encoder. SPL introduces three components: (1) swap-guided base regularization, (2) Preferential Inverse Autoregressive Flow (P-IAF), and (3) adaptive latent conditioning. Experiments show that SPL mitigates collapse, enriches user-specific latents, and improves preference prediction. Our code and data are available at https://github.com/cobang0111/SPL

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 36 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

28 of 36 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Latent Embedding Adaptation for Human Preference Alignment in Diffusion Planners24 Mar
  2. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning19 Aug
  3. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  5. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  6. 2015Variational Inference with Normalizing Flowscited by 2,170
  7. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,323
  8. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  9. 2015Generating Sentences from a Continuous Spacecited by 612
  10. 2015Importance Weighted Autoencoderscited by 456
  11. 2023AI Alignment: A Comprehensive Surveycited by 263
  12. 2024A Roadmap to Pluralistic Alignmentcited by 184
9 more
  1. 2016Variational Lossy Autoencodercited by 176
  2. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 126
  3. 2019Lagging Inference Networks and Posterior Collapse in Variational Autoencoderscited by 124
  4. 2019Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapsecited by 121
  5. 2023Posterior Collapse and Latent Variable Non-identifiabilitycited by 57
  6. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 38
  7. 2025Language Model Personalization via Reward Factorizationcited by 31
  8. 2023Moral Machine or Tyranny of the Majority?cited by 22
  9. 2025Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026AI Alignment From Social Choice Perspectivescited by 2
  2. 2026Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalizationcited by 1
  3. 2026Variance-aware Reward Modeling with Anchor Guidancecited by 0
  4. 2026Cautious Context Steering for Language Model Personalizationcited by 0
  5. 2026Using Context Is Not Enough: Test-Time Training for Personalized Reward Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning
  2. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
  3. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHF
  4. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  5. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  6. 2025When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF
  7. 2023Reinforcement Learning from Diverse Human Preferences
  8. 2026From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning