mlsanityData to 7 Oct 2026

Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward

arXiv 2601.19055 ↗First posted 27 Jan 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We study how to fine-tune LLMs using user-edit deployment data consisting of a set of context, an agent's response, and user edits. This deployment data is naturally generated by users in applications such as LLMs-based writing assistants and coding agents. The _natural_ origin of user edits makes it a desired source for adapting and personalizing LLMs. In this setup, there emerges a unification of various feedback types namely preferences, supervised labels, and cost that are typically studied separately in the literature. In this paper, we initiate the theoretical investigation of learning from user edits. We first derive bounds for learning algorithms that learn from each of these feedback types. We prove that these algorithms have different trade-offs depending upon the user, data distribution, and model class. We then propose a simple ensembling procedure to jointly learn from these feedback types. On two domains adapted from Gao et al. 2024, we show our ensembling procedure outperforms these methods that learn from individual feedback. Further, we show that our proposed procedure can robustly adapt to different user-edit distributions at test time.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 45 of its 61 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

45 of 61 references.

References · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  6. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  7. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  8. 2024Iterative Reasoning Preference Optimizationcited by 244
  9. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  10. 2024RepairAgent: An Autonomous, LLM-Based Agent for Program Repaircited by 225
  11. 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 200
  12. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 194
33 more
  1. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  2. 2018SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repaircited by 144
  3. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
  4. 2023An Analysis of the Automatic Bug Fixing Performance of ChatGPTcited by 108
  5. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  6. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  7. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  8. 2023Is RLHF More Difficult than Standard RL?cited by 96
  9. 2017Generating Sentences by Editing Prototypescited by 88
  10. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  11. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  12. 2024Aligning LLM Agents by Learning Latent Preference from User Editscited by 66
  13. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  14. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  15. 2020Felix: Flexible Text Editing Through Tagging and Insertioncited by 58
  16. 2020Seq2Edits: Sequence Transduction Using Span-level Edit Operationscited by 58
  17. 2023A Survey of Learning-based Automated Program Repaircited by 55
  18. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  19. 2022CoditT5: Pretraining for Source Code and Natural Language Editingcited by 45
  20. 2021LEWIS: Levenshtein Editing for Unsupervised Text Style Transfercited by 40
  21. 2022Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infillingcited by 38
  22. 2022On Improving Summarization Factual Consistency from Natural Language Feedbackcited by 38
  23. 2018Learning to Represent Editscited by 34
  24. 2023ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviewscited by 34
  25. 2022arXivEdits: Understanding the Human Revision Process in Scientific Writingcited by 26
  26. 2022An Imitation Learning Curriculum for Text Editing with Non-Autoregressive Modelscited by 20
  27. 2022Towards Automated Document Revision: Grammatical Error Correction, Fluency Edits, and Beyondcited by 20
  28. 2023User Modeling in the Era of Large Language Models: Current Research and Future Directionscited by 18
  29. 2024Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignmentcited by 9
  30. 2024Retrospective Learning from Interactionscited by 6
  31. 2020Variational Inference for Learning Representations of Natural Language Editscited by 5
  32. 2024Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerationscited by 5
  33. 2024PREDICT: Preference Reasoning by Evaluating Decomposed preferences Inferred from Candidate Trajectoriescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Fine-Tuning LLMs with Fine-Grained Human Feedback on Text Spans
  2. 2024A Framework for Fine-Tuning LLMs using Heterogeneous Feedback
  3. 2025Aligning LLMs by Predicting Preferences from User Writing Samples
  4. 2026Personalizing LLMs with Binary Feedback: A Preference-Corrected Optimization Framework
  5. 2026Goal-Conditioned Supervised Learning for LLM Fine-Tuning
  6. 2024PersonalLLM: Tailoring LLMs to Individual Preferences
  7. 2025User-Assistant Bias in LLMs
  8. 2024Orchestrating LLMs with Different Personalizations