mlsanityData to 14 Oct 2026

Preference-grounded Token-level Guidance for Language Model Fine-tuning

arXiv 2306.00398 ↗First posted 1 Jun 2023cs.CLcited by 31

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning language models (LMs) with preferences is an important problem in natural language generation. A key challenge is that preferences are typically provided at the sequence level while LM training and generation both occur at the token level. There is, therefore, a granularity mismatch between the preference and the LM training losses, which may complicate the learning problem. In this paper, we address this issue by developing an alternate training process, where we iterate between grounding the sequence-level preference into token-level training guidance, and improving the LM with the learned guidance. For guidance learning, we design a framework that extends the pairwise-preference learning in imitation learning to both variable-length LM generation and the utilization of the preference among multiple generations. For LM training, based on the amount of supervised data, we present two minimalist learning objectives that utilize the learned guidance. In experiments, our method performs competitively on two distinct representative LM tasks -- discrete-prompt generation and text summarization.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 94 of its 129 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

94 of 129 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Aligning Language Models with Preferences through f-divergence Minimization16 Feb
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 92

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  11. 2017Deep reinforcement learning from human preferencescited by 3,912
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
38 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  11. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  12. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  13. 2017Hindsight Experience Replaycited by 1,136
  14. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  15. 2015Teaching Machines to Read and Comprehendcited by 1,049
  16. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  17. 2021GPT Understands, Toocited by 716
  18. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  19. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  20. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  21. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  22. 2016Self-critical Sequence Training for Image Captioningcited by 547
  23. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  24. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 376
  25. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  26. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  27. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 358
  28. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  29. 2021True Few-Shot Learning with Language Modelscited by 344
  30. 2022Teaching language models to support answers with verified quotescited by 292
  31. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  32. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  33. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  34. 2020Critic Regularized Regressioncited by 275
  35. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  36. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 256
  37. 2023Pretraining Language Models with Human Preferencescited by 247
  38. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 12; 2026: 4.

31 papers.

Sort:

By area:

  1. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 127
  2. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  3. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  4. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  5. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  6. 2024Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraintcited by 39
  7. 2024Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learningcited by 33
  8. 2024Learning to Summarize from LLM-generated Feedbackcited by 24
  9. 2024FoRAG: Factuality-optimized Retrieval Augmented Generation for Web-enhanced Long-form Question Answeringcited by 18
  10. 2024Reinforcement Learning with Token-level Feedback for Controllable Text Generationcited by 16
  11. 2025Discriminative Policy Optimization for Token-Level Reward Modelscited by 16
  12. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
19 more
  1. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  2. 2023Let's Reinforce Step by Stepcited by 10
  3. 2024Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimizationcited by 7
  4. 2024Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuningcited by 5
  5. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Modelscited by 5
  6. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Modelscited by 5
  7. 2025Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preferencecited by 5
  8. 2025TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimizationcited by 5
  9. 2025Automatic LLM Red Teamingcited by 5
  10. 2025Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Datacited by 4
  11. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Controlcited by 4
  12. 2024Efficient Hybrid Inference for LLMs: Reward-Based Token Modelling with Selective Cloud Assistancecited by 2
  13. 2025Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMscited by 2
  14. 2025PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptionscited by 1
  15. 2026Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modelingcited by 1
  16. 2024Switchable Decision: Dynamic Neural Generation Networkscited by 0
  17. 2025References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulationcited by 0
  18. 2026ARCA: Adapter-Residual Credit Assignment When Token Signals Degeneratecited by 0
  19. 2026Robust Critics: Defending LLMs Against Multi-Turn Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Sequence-level Large Language Model Training with Contrastive Preference Optimization
  2. 2025PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training
  3. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  4. 2024Preference-Guided Reflective Sampling for Aligning Language Models
  5. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  6. 2023Training Language Models with Language Feedback at Scale
  7. 2024Aligning Large Language Models via Fine-grained Supervision
  8. 2024Aligning Language Models with Demonstrated Feedback