mlsanityData to 7 Oct 2026

Preference Alignment with Flow Matching

arXiv 2405.19806 ↗First posted 30 May 2024cs.LGcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We present Preference Flow Matching (PFM), a new framework for preference-based reinforcement learning (PbRL) that streamlines the integration of preferences into an arbitrary class of pre-trained models. Existing PbRL methods require fine-tuning pre-trained models, which presents challenges such as scalability, inefficiency, and the need for model modifications, especially with black-box APIs like GPT-4. In contrast, PFM utilizes flow matching techniques to directly learn from preference data, thereby reducing the dependency on extensive fine-tuning of pre-trained models. By leveraging flow-based models, PFM transforms less preferred data into preferred outcomes, and effectively aligns model outputs with human preferences without relying on explicit or implicit reward function estimation, thus avoiding common issues like overfitting in reward models. We provide theoretical insights that support our method's alignment with standard PbRL objectives. Experimental results indicate the practical effectiveness of our method, offering a new direction in aligning a pre-trained model to preference. Our code is available at https://github.com/jadehaus/preference-flow-matching.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 31 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

19 of 31 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Self-Rewarding Language Models18 Jan
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  3. 2023Contrastive Preference Learning: Learning from Human Feedback without RL20 Oct
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  6. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  7. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  8. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  9. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 9

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  6. 2023Improving and generalizing flow-based generative models with minibatch optimal transportcited by 835
  7. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  8. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  9. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 5.

11 papers.

  1. 2026Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Modelscited by 17
  2. 2025Mirror Flow Matching with Heavy-Tailed Priors for Generative Modeling on Convex Domainscited by 8
  3. 2024DeformPAM: Data-Efficient Learning for Long-horizon Deformable Object Manipulation via Preference-based Action Alignmentcited by 7
  4. 2025FlowRec: Prior-Informed Flow Matching for Efficient Sequential Recommendation Generationcited by 4
  5. 2025LAPP: Large Language Model Feedback for Preference-Driven Reinforcement Learningcited by 2
  6. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learningcited by 2
  7. 2025M3PO: Multimodal-Model-Guided Preference Optimization for Visual Instruction Followingcited by 1
  8. 2026FAVE: Flow-based Average Velocity Establishment for Sequential Recommendationcited by 1
  9. 2026Counterfactual Transport Flows for Offline Conservative Trajectory Refinementcited by 0
  10. 2026Tilted Schr"odinger Bridge Matchingcited by 0
  11. 2026Flow Matching Reinforcement for 3D Mesh Generation via Dynamic Homing Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026FlowCPO: A Unified Divergence View of Preference Alignment for Flow Models
  2. 2023Direct Preference-based Policy Optimization without Reward Modeling
  3. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learning
  4. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  5. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  6. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferences
  7. 2025Influencing Humans to Conform to Preference Models for RLHF
  8. 2026The Reward Was in Your Data All Along: Correcting Flow Matching with Discriminator-Guided RL