mlsanityData to 7 Oct 2026

ALaRM: Align Language Models via Hierarchical Rewards Modeling

arXiv 2403.06754 ↗First posted 11 Mar 2024cs.CL cs.AI cs.LGcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We introduce ALaRM, the first framework modeling hierarchical rewards in reinforcement learning from human feedback (RLHF), which is designed to enhance the alignment of large language models (LLMs) with human preferences. The framework addresses the limitations of current alignment approaches, which often struggle with the inconsistency and sparsity of human supervision signals, by integrating holistic rewards with aspect-specific rewards. This integration enables more precise and consistent guidance of language models towards desired outcomes, particularly in complex and open text generation tasks. By employing a methodology that filters and combines multiple rewards based on their consistency, the framework provides a reliable mechanism for improving model alignment. We validate our approach through applications in long-form question answering and machine translation tasks, employing gpt-3.5-turbo for pairwise comparisons, and demonstrate improvements over existing baselines. Our work underscores the effectiveness of hierarchical rewards modeling in refining LLM training processes for better human preference alignment. We release our code at https://ALaRM-fdu.github.io.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 52 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

44 of 52 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble30 Jan
  2. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  3. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2020Longformer: The Long-Document Transformercited by 3,362
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
28 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2023LIMA: Less Is More for Alignmentcited by 1,080
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  8. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  9. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  10. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  11. 2018AI safety via debatecited by 384
  12. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  13. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  14. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  15. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  16. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  17. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  18. 2018Supervising strong learners by amplifying weak expertscited by 139
  19. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  20. 2021Gender Bias in Machine Translationcited by 116
  21. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  22. 2017Video Captioning via Hierarchical Reinforcement Learningcited by 47
  23. 2017Ask the Right Questions: Active Question Reformulation with Reinforcement Learningcited by 44
  24. 2024Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Expertscited by 28
  25. 2019Hierarchical Reinforcement Learning for Open-Domain Dialogcited by 27
  26. 2018Hierarchical Reinforcement Learning with Hindsightcited by 26
  27. 2023Compositional preference models for aligning LMscited by 25
  28. 2022Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQLcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 1; 2026: 6.

11 papers.

  1. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  2. 2024MetaAligner: Towards Generalizable Multi-Objective Alignment of Language Modelscited by 43
  3. 2024HonestLLM: Toward an Honest and Helpful Large Language Modelcited by 17
  4. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  5. 2024HAF-RM: A Hybrid Alignment Framework for Reward Model Trainingcited by 5
  6. 2024CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedbackcited by 5
  7. 2026DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learningcited by 3
  8. 2026SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Modelscited by 3
  9. 2026GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimizationcited by 3
  10. 2026Reinforced Reasoning for End-to-End Retrosynthetic Planningcited by 0
  11. 2026ReCAST: Reward Credit Assignment across Timesteps for Online Diffusion Reinforcementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Rethinking the Role of Proxy Rewards in Language Model Alignment
  2. 2025Energy-Based Reward Models for Robust Language Model Alignment
  3. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  4. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  5. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
  6. 2025Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment
  7. 2025Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs
  8. 2023Aligning Large Language Models through Synthetic Feedback