mlsanityData to 7 Oct 2026

Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflection

arXiv 2403.14238 ↗First posted 21 Mar 2024cs.CL cs.AIcited by 7

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Despite the promise of RLHF in aligning LLMs with human preferences, it often leads to superficial alignment, prioritizing stylistic changes over improving downstream performance of LLMs. Underspecified preferences could obscure directions to align the models. Lacking exploration restricts identification of desirable outputs to improve the models. To overcome these challenges, we propose a novel framework: Reinforcement Learning from Reflective Feedback (RLRF), which leverages fine-grained feedback based on detailed criteria to improve the core capabilities of LLMs. RLRF employs a self-reflection mechanism to systematically explore and refine LLM responses, then fine-tuning the models via a RL algorithm along with promising responses. Our experiments across Just-Eval, Factuality, and Mathematical Reasoning demonstrate the efficacy and transformative potential of RLRF beyond superficial surface-level adjustment.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 51 references that are in the map. First posted on arXiv 21 Mar 2024.

Builds on

39 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Self-Rewarding Language Models18 Jan
  2. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  3. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  11. 2018Hierarchical Neural Story Generationcited by 1,182
  12. 2023LIMA: Less Is More for Alignmentcited by 1,080
20 more
  1. 2023Teaching Large Language Models to Self-Debugcited by 963
  2. 2022Large Language Models Can Self-Improvecited by 656
  3. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  4. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  5. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  6. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  7. 2022Generating Sequences by Learning to Self-Correctcited by 271
  8. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  9. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  10. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  11. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  12. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  13. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  14. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  15. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 103
  16. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  17. 2022The CRINGE Loss: Learning what language not to modelcited by 39
  18. 2024Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraintcited by 39
  19. 2023Automatic Model Selection with Large Language Models for Reasoningcited by 35
  20. 2024Small Language Model Can Self-correctcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2026Reinforcement Learning via Self-Distillationcited by 285
  2. 2024Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMscited by 22
  3. 2026Aligning Language Models from User Interactionscited by 21
  4. 2025KnowRL: Teaching Language Models to Know What They Knowcited by 5
  5. 2024Learning to Reason via Self-Iterative Process Feedback for Small Language Modelscited by 2
  6. 2025MathSE: Improving Multimodal Mathematical Reasoning via Self-Evolving Iterative Reflection and Reward-Guided Fine-Tuningcited by 2
  7. 2025TASER: Table Agents for Schema-guided Extraction and Recommendationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedback
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  3. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  4. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
  5. 2025Reflective Preference Optimization (RPO): Enhancing On-Policy Alignment via Hint-Guided Reflection
  6. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
  7. 2026ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework
  8. 2023Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond