mlsanityData to 7 Oct 2026

Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language

arXiv 2311.14543 ↗First posted 24 Nov 2023cs.CL cs.AIcited by 22

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Learning from human feedback is a prominent technique to align the output of large language models (LLMs) with human expectations. Reinforcement learning from human feedback (RLHF) leverages human preference signals that are in the form of ranking of response pairs to perform this alignment. However, human preference on LLM outputs can come in much richer forms including natural language, which may provide detailed feedback on strengths and weaknesses of a given response. In this work we investigate data efficiency of modeling human feedback that is in natural language. Specifically, we fine-tune an open-source LLM, e.g., Falcon-40B-Instruct, on a relatively small amount (1000 records or even less) of human feedback in natural language in the form of critiques and revisions of responses. We show that this model is able to improve the quality of responses from even some of the strongest LLMs such as ChatGPT, BARD, and Vicuna, through critique and revision of those responses. For instance, through one iteration of revision of ChatGPT responses, the revised responses have 56.6% win rate over the original ones, and this win rate can be further improved to 65.9% after applying the revision for five iterations.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 32 references that are in the map. First posted on arXiv 24 Nov 2023.

Builds on

27 of 32 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Preference Ranking Optimization for Human Alignment30 Jun
  2. 2023Aligning Large Language Models through Synthetic Feedback23 May
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  9. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  10. 2022Self-critiquing models for assisting human evaluatorscited by 372
  11. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  12. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
11 more
  1. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  2. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  3. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  4. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  5. 2022PEER: A Collaborative Language Modelcited by 86
  6. 2022Training Language Models with Language Feedbackcited by 48
  7. 2022Learning to Model Editing Processescited by 30
  8. 2022EditEval: An Instruction-Based Benchmark for Text Improvementscited by 30
  9. 2022Text Generation with Text-Editing Modelscited by 21
  10. 2023Small Language Models Improve Giants by Rewriting Their Outputscited by 18
  11. 2022Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systemscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 11; 2026: 3.

22 papers.

Sort:

By area:

  1. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 94
  2. 2024From Exploration to Mastery: Enabling LLMs to Master Tools via Self-Driven Interactionscited by 46
  3. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  4. 2025Agent4Edu: Generating Learner Response Data by Generative Agents for Intelligent Education Systemscited by 32
  5. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approachcited by 24
  6. 2024AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrationscited by 19
  7. 2024Naturally Occurring Feedback is Common, Extractable and Usefulcited by 14
  8. 2024From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verificationcited by 8
  9. 2025Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behaviorcited by 8
  10. 2025Modeling Challenging Patient Interactions: LLMs for Medical Communication Trainingcited by 7
  11. 2025FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Datacited by 5
  12. 2024The Real, the Better: Aligning Large Language Models with Online Human Behaviorscited by 3
10 more
  1. 2025Empowering Economic Simulation for Massively Multiplayer Online Games through Generative Agent-Based Modelingcited by 3
  2. 2025Beyond Sample-Level Feedback: Using Reference-Level Feedback to Guide Data Synthesiscited by 2
  3. 2025Towards AI-Driven Human-Machine Co-Teaming for Adaptive and Agile Cyber Security Operation Centerscited by 2
  4. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1
  5. 2025GOD model: Privacy Preserved AI School for Personal Assistantcited by 1
  6. 2025Computing Voting Rules with Improvement Feedbackcited by 0
  7. 2025Textual Self-attention Network: Test-Time Preference Optimization through Textual Gradient-based Attentioncited by 0
  8. 2026See2Refine: Vision-Language Feedback Improves LLM-Based eHMI Action Designerscited by 0
  9. 2026Efficiently Aligning Language Models with Online Natural Language Feedbackcited by 0
  10. 2026Generating Natural and Expressive Robot Gestures through Iterative Reinforcement Learning with Human Feedback using LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  2. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  3. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  4. 2025DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decomposition
  5. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment
  6. 2024Aligning Large Language Models via Fine-grained Supervision
  7. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback
  8. 2023Aligning Language Models with Offline Learning from Human Feedback