Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spans
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
We present a method and dataset for fine-tuning language models with preference supervision using feedback-driven improvement chains. Given a model response, an annotator provides fine-grained feedback by marking ``liked'' and ``disliked'' spans and specifying what they liked or disliked about them. The base model then rewrites the disliked spans accordingly, proceeding from left to right, forming a sequence of incremental improvements. We construct preference pairs for direct alignment from each adjacent step in the chain, enabling the model to learn from localized, targeted edits. We find that our approach outperforms direct alignment methods based on standard A/B preference ranking or full contrastive rewrites, demonstrating that structured, revision-based supervision leads to more efficient and effective preference tuning.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 28 references that are in the map. First posted on arXiv 29 Dec 2025.
Builds on
21 of 28 references.
Closest ancestors · 9
Same direction, newest first
- 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization24 Feb
- 2024Self-Boosting Large Language Models with Synthetic Preference Data9 Oct
- 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment12 Aug
- 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment19 Jul
- 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferences12 Mar
- 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
- 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
- 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment7 Nov
- 2022Training language models to follow instructions with human feedback4 Mar
Other references · 12
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
- 2022Large Language Models Can Self-Improvecited by 656
- 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
- 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
- 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
- 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
- 2024Improving Reward Models with Synthetic Critiquescited by 44
- 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
- 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computationcited by 24
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Aligning Large Language Models via Fine-grained Supervision
- 2025Refine-n-Judge: Curating High-Quality Preference Chains for LLM-Fine-Tuning
- 2024Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process
- 2024A Framework for Fine-Tuning LLMs using Heterogeneous Feedback
- 2023ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference
- 2026Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward
- 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
- 2026Goal-Conditioned Supervised Learning for LLM Fine-Tuning