mlsanityData to 7 Oct 2026

On the Transformations across Reward Model, Parameter Update, and In-Context Prompt

arXiv 2406.16377 ↗First posted 24 Jun 2024cs.CL cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Despite the general capabilities of pre-trained large language models (LLMs), they still need further adaptation to better serve practical applications. In this paper, we demonstrate the interchangeability of three popular and distinct adaptation tools: parameter updating, reward modeling, and in-context prompting. This interchangeability establishes a triangular framework with six transformation directions, each of which facilitates a variety of applications. Our work offers a holistic view that unifies numerous existing studies and suggests potential research directions. We envision our work as a useful roadmap for future research on LLMs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 110 of its 147 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

110 of 147 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024RLVF: Learning from Verbal Feedback without Overgeneralization16 Feb
  2. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF11 Feb
  3. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  4. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  5. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  6. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  10. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followers10 Feb
  11. 2023Chain of Hindsight Aligns Language Models with Feedback6 Feb
  12. 2022Scaling Laws for Reward Model Overoptimization19 Oct

Other references · 98

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  2. 2023Let's Verify Step by Stepcited by 3,598
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  11. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  12. 2023LIMA: Less Is More for Alignmentcited by 1,080
  13. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  14. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  15. 2016HyperNetworkscited by 1,003
  16. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  17. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  18. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  19. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  20. 2022Transformers learn in-context by gradient descentcited by 714
  21. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  22. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  23. 2022Large Language Models Can Self-Improvecited by 656
  24. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  25. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  26. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  27. 2021Editing Factual Knowledge in Language Modelscited by 548
  28. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  29. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  30. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  31. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  32. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  33. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  34. 2023Larger language models do in-context learning differentlycited by 386
  35. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  36. 2022Self-critiquing models for assisting human evaluatorscited by 372
  37. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  38. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321

50 most cited of 98 All 98 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Promptscited by 9

Similar, not cited

Close in wording, no citation either way

  1. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings
  2. 2024Transforming and Combining Rewards for Aligning Large Language Models
  3. 2024Self-Instructed Derived Prompt Generation Meets In-Context Learning: Unlocking New Potential of Black-Box LLMs
  4. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  5. 2024The Power of Adaptation: Boosting In-Context Learning through Adaptive Prompting
  6. 2024A critical review of methods and challenges in large language models
  7. 2025Cross-Lingual Prompt Steerability: Towards Accurate and Robust LLM Behavior across Languages
  8. 2023Reward Collapse in Aligning Large Language Models