mlsanityData to 7 Oct 2026

TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching

arXiv 2605.12288 ↗First posted 12 May 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) is a widely used RL-free method for aligning language models from pairwise preferences, but it models preferences over full sequences even though generation is driven by per-token decisions. Existing token-level extensions typically decompose a sequence-level Bradley-Terry objective across timesteps, leaving per-prefix (state-wise) optimality implicit. We study how to recover token-level preference optimality using only standard sequence-level pairwise comparisons. We introduce Token-level Bregman Preference Optimization (TBPO), which posits a token-level Bradley-Terry preference model over next-token actions conditioned on the prefix, and derive a Bregman-divergence density-ratio matching objective that generalizes the logistic/DPO loss while preserving the optimal policy induced by the token-level model and maintaining DPO-like simplicity. We introduce two instantiations: TBPO-Q, which explicitly learns a lightweight state baseline, and TBPO-A, which removes the baseline through advantage normalization. Across instruction following, helpfulness/harmlessness, and summarization benchmarks, TBPO improves alignment quality and training stability and increases output diversity relative to strong sequence-level and token-level baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 52 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

30 of 52 references.

Closest ancestors · 13

Same direction, newest first

  1. 2026MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation2 May
  2. 2026SRA: Span Representation Alignment for Large Language Model Distillation2 May
  3. 2026Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimization18 Mar
  4. 2026CTPD: Cross Tokenizer Preference Distillation17 Jan
  5. 2025Preference Optimization by Estimating the Ratio of the Data Distribution26 May
  6. 2025A Survey of Direct Preference Optimization12 Mar
  7. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights6 Oct
  8. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  10. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  11. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
1 more
  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 17

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  9. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  10. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  11. 2022Concrete Score Matching: Generalized Score Matching for Discrete Datacited by 144
  12. 2025Judge's Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreementcited by 24
5 more
  1. 2025CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewardscited by 11
  2. 2025XTRA: Cross-Lingual Topic Modeling with Topic and Representation Alignmentscited by 3
  3. 2026GloCTM: Cross-Lingual Topic Modeling via a Global Context Spacecited by 3
  4. 2026MIPIC: Matryoshka Representation Learning via Self-Distilled Intra-Relational and Progressive Information Chainingcited by 2
  5. 2026LLM-XTM: Enhancing Cross-Lingual Topic Models with Large Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Selective Preference Optimization via Token-Level Reward Function Estimation
  2. 2025ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
  3. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  4. 2025Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective
  5. 2025Token-Importance Guided Direct Preference Optimization
  6. 2025Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization
  7. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  8. 2026A Zeroth-Order Paradigm for LLM Preference Alignment