mlsanityData to 13 Oct 2026

Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment

arXiv 2311.04072 ↗First posted 7 Nov 2023cs.CLcited by 35

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Alignment with human preference is a desired property of large language models (LLMs). Currently, the main alignment approach is based on reinforcement learning from human feedback (RLHF). Despite the effectiveness of RLHF, it is intricate to implement and train, thus recent studies explore how to develop alternative alignment approaches based on supervised fine-tuning (SFT). A major limitation of SFT is that it essentially does imitation learning, which cannot fully understand what are the expected behaviors. To address this issue, we propose an improved alignment approach named FIGA. Different from prior methods, we incorporate fine-grained (i.e., token or phrase level) quality signals that are derived by contrasting good and bad responses. Our approach has made two major contributions. Firstly, we curate a refined alignment dataset that pairs initial responses and the corresponding revised ones. Secondly, we devise a new loss function can leverage fine-grained quality signals to instruct the learning of LLMs for alignment. Extensive experiments have demonstrated the effectiveness of our approaches by comparing a number of competitive baselines.

Key facts. Cited by 35 papers in the map (data through 13 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 51 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

40 of 51 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  2. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  3. 2023Fundamental Limitations of Alignment in Large Language Models19 Apr
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2017Proximal Policy Optimization Algorithmscited by 14,592
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,348
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,765
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  12. 2023Let's Verify Step by Stepcited by 3,672
24 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,500
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,007
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  4. 2023A Survey of Large Language Modelscited by 2,559
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,120
  6. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 705
  7. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 680
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 617
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 492
  10. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 459
  11. 2018Gender Bias in Coreference Resolutioncited by 439
  12. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 400
  13. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 377
  14. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  15. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  16. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  17. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 222
  18. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  19. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 127
  20. 2019Bridging the Gap between Training and Inference for Neural Machine Translationcited by 123
  21. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  22. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 60
  23. 2023Second Thoughts are Best: Learning to Re-Align With Human Values from Text Editscited by 27
  24. 2023Reward Collapse in Aligning Large Language Modelscited by 24

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 19; 2025: 13; 2026: 3.

35 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 279
  2. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 172
  3. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 81
  4. 2024Model Merging and Safety Alignment: One Bad Model Spoils the Bunchcited by 43
  5. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  6. 2024Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraintcited by 39
  7. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedbackcited by 25
  8. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24
  9. 2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 19
  10. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  11. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillationcited by 15
  12. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
23 more
  1. 2024Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillationcited by 14
  2. 2024Aligning Large Language Models via Fine-grained Supervisioncited by 13
  3. 2024T-REG: Preference Optimization with Token-Level Reward Regularizationcited by 13
  4. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  5. 2024KnowTuning: Knowledge-aware Fine-tuning for Large Language Modelscited by 11
  6. 2024Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappingscited by 11
  7. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  8. 2025ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimizationcited by 10
  9. 2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10
  10. 2025From Generic Empathy to Personalized Emotional Support: A Self-Evolution Framework for User Preference Alignmentcited by 9
  11. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Taskscited by 8
  12. 2024Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimizationcited by 7
  13. 2024Token-level Proximal Policy Optimization for Query Generationcited by 7
  14. 20242D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervisioncited by 6
  15. 2025Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillationcited by 6
  16. 2025Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preferencecited by 5
  17. 2025DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimizationcited by 3
  18. 2025Learning Explainable Dense Reward Shapes via Bayesian Optimizationcited by 2
  19. 2025Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimizationcited by 1
  20. 2025Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spanscited by 1
  21. 2026Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modelingcited by 1
  22. 2025Factors That Support Grounded Responses in LLM Conversations: A Rapid Reviewcited by 0
  23. 2026PLOT: Enhancing Preference Learning via Optimal Transportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Supervised Fine-Tuning as Inverse Reinforcement Learning
  2. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning
  3. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
  4. 2025Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only
  5. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment
  6. 2026DEFT: Distribution-guided Efficient Fine-Tuning for Human Alignment
  7. 2026Alignment Dynamics in LLM Fine-Tuning
  8. 2023Aligning Large Language Models through Synthetic Feedback