# Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models

arXiv: https://arxiv.org/abs/2411.08733 · HTML: https://mlsanity.com/paper/2411.08733/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 33 of its 74 references that are in the map.
- First posted on arXiv 13 Nov 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Learning To Retrieve Prompts for In-Context Learning](https://mlsanity.com/paper/2112.08633/) (2021)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision](https://mlsanity.com/paper/2312.09390/) (2023)
- [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://mlsanity.com/paper/2305.03047/) (2023)
- [PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization](https://mlsanity.com/paper/2310.16427/) (2023)
- [ReFT: Representation Finetuning for Language Models](https://mlsanity.com/paper/2404.03592/) (2024)
- [RAIN: Your Language Models Can Align Themselves without Finetuning](https://mlsanity.com/paper/2309.07124/) (2023)
- [ARGS: Alignment as Reward-Guided Search](https://mlsanity.com/paper/2402.01694/) (2024)
- [InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance](https://mlsanity.com/paper/2401.11206/) (2024)
- [Aligning Large Language Models through Synthetic Feedback](https://mlsanity.com/paper/2305.13735/) (2023)
- [Aligning Large Language Models with Representation Editing: A Control Perspective](https://mlsanity.com/paper/2406.05954/) (2024)
- [Towards Scalable Automated Alignment of LLMs: A Survey](https://mlsanity.com/paper/2406.01252/) (2024)
- [Human-Instruction-Free LLM Self-Alignment with Limited Samples](https://mlsanity.com/paper/2401.06785/) (2024)

## Built on by (most cited first, 13 of 13)

- [What Makes a Good Natural Language Prompt?](https://mlsanity.com/paper/2506.06950/) (2025)
- [CritiQ: Mining Data Quality Criteria from Human Preferences](https://mlsanity.com/paper/2502.19279/) (2025)
- [AdAEM: An Adaptively and Automated Extensible Measurement of LLMs' Value Difference](https://mlsanity.com/paper/2505.13531/) (2025)
- [Structured Prompts Improve Evaluation of Language Models](https://mlsanity.com/paper/2511.20836/) (2025)
- [See Further, Think Deeper: Advancing VLM's Reasoning Ability with Low-level Visual Cues and Reflection](https://mlsanity.com/paper/2604.24339/) (2026)
- [LLMdoctor: Token-Level Flow-Guided Preference Optimization for Efficient Test-Time Alignment of Large Language Models](https://mlsanity.com/paper/2601.10416/) (2026)
- [RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models](https://mlsanity.com/paper/2603.00724/) (2026)
- [SI-Agent: An Agentic Framework for Feedback-Driven Generation and Tuning of Human-Readable System Instructions for Large Language Models](https://mlsanity.com/paper/2507.03223/) (2025)
- [Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Models](https://mlsanity.com/paper/2508.05165/) (2025)
- [Light Alignment Improves LLM Safety via Model Self-Reflection with a Single Neuron](https://mlsanity.com/paper/2602.02027/) (2026)
- [Wait, am I Being Fair? Characterizing Deductive Stereotyping and Mitigating It with Fair-GCG](https://mlsanity.com/paper/2606.30989/) (2026)
- [When LLM Meets Tree Search: A Systematic View of Inference as Search in Large Language Models](https://mlsanity.com/paper/2608.30395/) (2026)
- [PrimeScientist: Strategic Allocation of Research Effort in Autonomous Research](https://mlsanity.com/paper/2609.17846/) (2026)

## Abstract (arXiv)

Aligning Large Language Models (LLMs) traditionally relies on costly training and human preference annotations. Self-alignment seeks to reduce these expenses by enabling models to align themselves. To further lower costs and achieve alignment without any expensive tuning or annotations, we introduce a new tuning-free approach for self-alignment, Dynamic Rewarding with Prompt Optimization (DRPO). Our approach leverages a search-based optimization framework that allows LLMs to iteratively self-improve and craft the optimal alignment instructions, all without additional training or human intervention. The core of DRPO is a dynamic rewarding mechanism, which identifies and rectifies model-specific alignment weaknesses, allowing LLMs to adapt efficiently to diverse alignment challenges. Empirical evaluations on eight recent LLMs, both open- and closed-sourced, demonstrate that DRPO significantly enhances alignment performance, with base models outperforming their SFT/RLHF-tuned counterparts. Moreover, the prompts automatically optimized by DRPO surpass those curated by human experts, further validating the effectiveness of our approach. Our findings highlight the great potential of current LLMs to achieve adaptive self-alignment through inference-time optimization, complementing tuning-based alignment methods.