mlsanityData to 14 Oct 2026

The Real, the Better: Aligning Large Language Models with Online Human Behaviors

arXiv 2405.00578 ↗First posted 1 May 2024cs.CL cs.AIcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language model alignment is widely used and studied to avoid LLM producing unhelpful and harmful responses. However, the lengthy training process and predefined preference bias hinder adaptation to online diverse human preferences. To this end, this paper proposes an alignment framework, called Reinforcement Learning with Human Behavior (RLHB), to align LLMs by directly leveraging real online human behaviors. By taking the generative adversarial framework, the generator is trained to respond following expected human behavior; while the discriminator tries to verify whether the triplets of query, response, and human behavior come from real online environments. Behavior modeling in natural-language form and the multi-model joint training mechanism enable an active and sustainable online alignment. Experimental results confirm the effectiveness of our proposed methods by both human and automatic evaluations.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 48 references that are in the map. First posted on arXiv 1 May 2024.

Builds on

32 of 48 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  2. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language24 Nov
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2023RAIN: Your Language Models Can Align Themselves without Finetuning13 Sep
  5. 2023Preference Ranking Optimization for Human Alignment30 Jun
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 26

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2016Generative Adversarial Imitation Learningcited by 1,597
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  10. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  12. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
14 more
  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  2. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  3. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  4. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  5. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  6. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  7. 2020Embedding-based Retrieval in Facebook Searchcited by 156
  8. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
  9. 2023Shepherd: A Critic for Language Model Generationcited by 97
  10. 2023WebCPM: Interactive Web Search for Chinese Long-form Question Answeringcited by 93
  11. 2023Learning to Generate Better Than Your LLMcited by 49
  12. 2023Safer-Instruct: Aligning Language Models with Automated Preference Datacited by 27
  13. 2021FeedRec: News Feed Recommendation with Various User Feedbackscited by 23
  14. 2023LaFFi: Leveraging Hybrid Natural Language Feedback for Fine-tuning Language Modelscited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Understanding Layer Significance in LLM Alignmentcited by 19
  2. 2025Assessing the Potential of Generative Agents in Crowdsourced Fact-Checkingcited by 3
  3. 2025R3A: Reinforced Reasoning for Relevance Assessment for RAG in User-Generated Content Platformscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  2. 2023Aligning Large Language Models through Synthetic Feedback
  3. 2025RLTHF: Targeted Human Feedback for LLM Alignment
  4. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  5. 2025RLHF: A comprehensive Survey for Cultural, Multimodal and Low Latency Alignment Methods
  6. 2024Understanding the Learning Dynamics of Alignment with Human Feedback
  7. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  8. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback