mlsanityData to 7 Oct 2026

Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision

arXiv 2403.09472 ↗First posted 14 Mar 2024cs.LG cs.AI cs.CLcited by 95

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Current AI alignment methodologies rely on human-provided demonstrations or judgments, and the learned capabilities of AI systems would be upper-bounded by human capabilities as a result. This raises a challenging research question: How can we keep improving the systems when their capabilities have surpassed the levels of humans? This paper answers this question in the context of tackling hard reasoning tasks (e.g., level 4-5 MATH problems) via learning from human annotations on easier tasks (e.g., level 1-3 MATH problems), which we term as easy-to-hard generalization. Our key insight is that an evaluator (reward model) trained on supervisions for easier tasks can be effectively used for scoring candidate solutions of harder tasks and hence facilitating easy-to-hard generalization over different levels of tasks. Based on this insight, we propose a novel approach to scalable alignment, which firstly trains the (process-supervised) reward models on easy problems (e.g., level 1-3), and then uses them to evaluate the performance of policy models on hard problems. We show that such easy-to-hard generalization from evaluators can enable easy-to-hard generalizations in generators either through re-ranking or reinforcement learning (RL). Notably, our process-supervised 7b RL model and 34b model (reranking@1024) achieves an accuracy of 34.0% and 52.5% on MATH500, respectively, despite only using human supervision on easy problems. Our approach suggests a promising path toward AI systems that advance beyond the frontier of human supervision.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 80 of its 103 references that are in the map. First posted on arXiv 14 Mar 2024.

Builds on

80 of 103 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision14 Dec
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback22 May
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec
  4. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  5. 2020Learning to summarize from human feedback2 Sep

Other references · 75

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  12. 2016Concrete Problems in AI Safetycited by 1,582
  13. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  14. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  15. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  16. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  17. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  18. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  19. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  20. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  21. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  22. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  23. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  24. 2021Measuring Coding Challenge Competence With APPScited by 879
  25. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  26. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  27. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  28. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  29. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  30. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  31. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  32. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  33. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  34. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  35. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  36. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  37. 2023Llemma: An Open Language Model For Mathematicscited by 385
  38. 2020Generative Language Modeling for Automated Theorem Provingcited by 379

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 37; 2025: 42; 2026: 16.

95 papers.

Sort:

By area:

  1. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  2. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  3. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  4. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  5. 2024AI Agents That Mattercited by 199
  6. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  7. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  8. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  9. 2025Process Reward Models That Thinkcited by 118
  10. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  11. 2024LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoningcited by 90
  12. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
38 more
  1. 2024Can Language Models Learn to Skip Steps?cited by 59
  2. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  3. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  4. 2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoningcited by 41
  5. 2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 40
  6. 2024MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoningcited by 40
  7. 2025VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Datacited by 39
  8. 2025R-PRM: Reasoning-Driven Process Reward Modelingcited by 38
  9. 2024AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinementcited by 33
  10. 2025Optimizing Temperature for Language Models with Multi-Sample Inferencecited by 33
  11. 2024Weak-to-Strong Reasoningcited by 32
  12. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  13. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  14. 2024Diving into Self-Evolving Training for Multimodal Reasoningcited by 31
  15. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
  16. 2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignmentcited by 26
  17. 2024B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasonerscited by 26
  18. 2025Retrieval-Augmented Process Reward Model for Generalizable Mathematical Reasoningcited by 23
  19. 2024Evaluating Robustness of Reward Models for Mathematical Reasoningcited by 21
  20. 2024Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlocited by 21
  21. 2024Weak-to-Strong Generalization Through the Data-Centric Lenscited by 21
  22. 2024Your Weak LLM is Secretly a Strong Teacher for Alignmentcited by 20
  23. 2025Weak-for-Strong: Training Weak Meta-Agent to Harness Strong Executorscited by 20
  24. 2025video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Modelcited by 19
  25. 2024HonestLLM: Toward an Honest and Helpful Large Language Modelcited by 17
  26. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  27. 2025Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifierscited by 14
  28. 2025MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environmentscited by 13
  29. 2026ToolPRMBench: Evaluating and Advancing Process Reward Models for Tool-using Agentscited by 13
  30. 2026Not All Queries Need Deep Thought: CoFiCot for Adaptive Coarse-to-fine Stateful Refinementcited by 13
  31. 2024MathGAP: Out-of-Distribution Evaluation on Problems with Arbitrarily Complex Proofscited by 12
  32. 2025Sample Complexity and Representation Ability of Test-time Scaling Paradigmscited by 12
  33. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 11
  34. 2025Discrepancies are Virtue: Weak-to-Strong Generalization through Lens of Intrinsic Dimensioncited by 10
  35. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimizationcited by 10
  36. 2024Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?cited by 9
  37. 2024Varying Shades of Wrong: Aligning LLMs with Wrong Answers Onlycited by 9
  38. 2025Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMscited by 9

50 most cited of 95 All 95 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models
  2. 2024Explanation, Debate, Align: A Weak-to-Strong Framework for Language Model Generalization
  3. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learning
  4. 2024Improving Weak-to-Strong Generalization with Scalable Oversight and Ensemble Learning
  5. 2025Intrinsic Barriers and Practical Pathways for Human-AI Alignment: An Agreement-Based Complexity Analysis
  6. 2026Limits of Difficulty Scaling: Hard Samples Yield Diminishing Returns in GRPO-Tuned SLMs
  7. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence
  8. 2025Position: The Complexity of Perfect AI Alignment -- Formalizing the RLHF Trilemma