mlsanityData to 7 Oct 2026

DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization

arXiv 2501.03271 ↗First posted 5 Jan 2025cs.LG cs.AI cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The rapid rise of large language models (LLMs) has unlocked many applications but also underscores the challenge of aligning them with diverse values and preferences. Direct Preference Optimization (DPO) is central to alignment but constrained by fixed divergences and limited feature transformations. We propose DPO-Kernels, which integrates kernel methods to address these issues through four key contributions: (i) Kernelized Representations with polynomial, RBF, Mahalanobis, and spectral kernels for richer transformations, plus a hybrid loss combining embedding-based and probability-based objectives; (ii) Divergence Alternatives (Jensen-Shannon, Hellinger, Renyi, Bhattacharyya, Wasserstein, and f-divergences) for greater stability; (iii) Data-Driven Selection metrics that automatically choose the best kernel-divergence pair; and (iv) a Hierarchical Mixture of Kernels for both local precision and global modeling. Evaluations on 12 datasets demonstrate state-of-the-art performance in factuality, safety, reasoning, and instruction following. Grounded in Heavy-Tailed Self-Regularization, DPO-Kernels maintains robust generalization for LLMs, offering a comprehensive resource for further alignment research.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 111 references that are in the map. First posted on arXiv 5 Jan 2025.

Builds on

30 of 111 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback13 Jun
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 28

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  12. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
16 more
  1. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  2. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  3. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  5. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  6. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  7. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  8. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  9. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  10. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  11. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 138
  12. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 125
  13. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 54
  14. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Modelscited by 34
  15. 2024RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Datacited by 15
  16. 2023DiffuSIA: A Spiral Interaction Architecture for Encoder-Decoder Text Diffusioncited by 1

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025A Survey of Direct Preference Optimizationcited by 35
  2. 2025Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Less is More: Improving LLM Alignment via Preference Data Selection
  2. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
  3. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
  4. 2024sDPO: Don't Use Your Data All at Once
  5. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization
  6. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forward
  7. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
  8. 2026SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalization