mlsanityData to 14 Oct 2026

Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint

arXiv 2401.06081 ↗First posted 11 Jan 2024cs.CLcited by 39

Not on the map. Why?

Cited by 20 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reinforcement learning (RL) has been widely used in training large language models (LLMs) for preventing unexpected outputs, eg reducing harmfulness and errors. However, existing RL methods mostly adopt the instance-level reward, which is unable to provide fine-grained supervision for complex reasoning tasks, and can not focus on the few key tokens that lead to the incorrectness. To address it, we propose a new RL method named RLMEC that incorporates a generative model as the reward model, which is trained by the erroneous solution rewriting task under the minimum editing constraint, and can produce token-level rewards for RL training. Based on the generative reward model, we design the token-level RL objective for training and an imitation-based regularization for stabilizing RL process. And the both objectives focus on the learning of the key tokens for the erroneous solution, reducing the effect of other unimportant tokens. The experiment results on mathematical tasks and question-answering tasks have demonstrated the effectiveness of our approach. Our code and data are available at https://github.com/RUCAIBox/RLMEC.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 47 of its 58 references that are in the map. First posted on arXiv 11 Jan 2024.

Builds on

47 of 58 references.

References · 47

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
35 more
  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  12. 2015Teaching Machines to Read and Comprehendcited by 1,049
  13. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  14. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  15. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  16. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  17. 2020Aligning AI With Shared Human Valuescited by 723
  18. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  19. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  20. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  21. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 343
  22. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  23. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 252
  24. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 237
  25. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  26. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 182
  27. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  28. 2022Successive Prompting for Decomposing Complex Questionscited by 94
  29. 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 82
  30. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 74
  31. 2023Making Large Language Models Better Reasoners with Alignmentcited by 63
  32. 2023Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoningcited by 49
  33. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignmentcited by 35
  34. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuningcited by 31
  35. 2022JiuZhang: A Chinese Pre-trained Language Model for Mathematical Problem Understandingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 14; 2026: 6.

39 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  2. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  3. 2025Multiagent Finetuning: Self Improvement with Diverse Reasoning Chainscited by 95
  4. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  5. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  6. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  7. 2025SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoningcited by 44
  8. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  9. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 39
  10. 2024Direct Judgement Preference Optimizationcited by 33
  11. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  12. 2024Self-rationalization improves LLM as a fine-grained judgecited by 27
27 more
  1. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24
  2. 2025Text2Grad: Reinforcement Learning from Natural Language Feedbackcited by 18
  3. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillationcited by 17
  4. 2024Reinforcing Language Agents via Policy Optimization with Action Decompositioncited by 16
  5. 2024Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineeringcited by 16
  6. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  7. 2024T-REG: Preference Optimization with Token-Level Reward Regularizationcited by 13
  8. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcementcited by 12
  9. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  10. 2024KnowTuning: Knowledge-aware Fine-tuning for Large Language Modelscited by 11
  11. 2025Spontaneous Giving and Calculated Greed in Language Modelscited by 10
  12. 2024On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigationcited by 9
  13. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedbackcited by 9
  14. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9
  15. 2024Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflectioncited by 8
  16. 2024Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignmentcited by 7
  17. 2024Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimizationcited by 7
  18. 20242D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervisioncited by 6
  19. 2025IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluationcited by 4
  20. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 4
  21. 2025Flipping Knowledge Distillation: Leveraging Small Models' Expertise to Enhance LLMs in Text Matchingcited by 3
  22. 2025Learning Explainable Dense Reward Shapes via Bayesian Optimizationcited by 2
  23. 2026Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Modelscited by 2
  24. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  25. 2025Pastiche Novel Generation Creating: Fan Fiction You Love in Your Favorite Author's Stylecited by 1
  26. 2026Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modelingcited by 1
  27. 2026Difference Feedback: Generating Multimodal Process-Level Supervision for VLM Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  3. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning
  4. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  5. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  6. 2025Token-Efficient RL for LLM Reasoning
  7. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example
  8. 2025Reinforcement Pre-Training