mlsanityData to 7 Oct 2026

ZYN: Zero-Shot Reward Models with Yes-No Questions for RLAIF

arXiv 2308.06385 ↗First posted 11 Aug 2023cs.CL cs.AIcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

In this work, we address the problem of directing the text generation of a language model (LM) towards a desired behavior, aligning the generated text with the preferences of the human operator. We propose using another, instruction-tuned language model as a critic reward model in a zero-shot way thanks to the prompt of a Yes-No question that represents the user preferences, without requiring further labeled data. This zero-shot reward model provides the learning signal to further fine-tune the base LM using Reinforcement Learning from AI Feedback (RLAIF); yet our approach is also compatible in other contexts such as quality-diversity search. Extensive evidence of the capabilities of the proposed ZYN framework is provided through experiments in different domains related to text generation, including detoxification; optimizing sentiment of movie reviews, or any other attribute; steering the opinion about a particular topic the model may have; and personalizing prompt generators for text-to-image tasks. Code available at urlhttps://github.com/vicgalle/zero-shot-reward-models/.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 55 references that are in the map. First posted on arXiv 11 Aug 2023.

Builds on

39 of 55 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  3. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2020Learning to summarize from human feedbackcited by 2,562
24 more
  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  5. 2021Ethical and social risks of harm from Language Modelscited by 957
  6. 2023Whose Opinions Do Language Models Reflect?cited by 759
  7. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  8. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  9. 2022Self-critiquing models for assisting human evaluatorscited by 372
  10. 2022Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLPcited by 316
  11. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  12. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  13. 2022Defining and Characterizing Reward Hackingcited by 184
  14. 2021Alignment of Language Agentscited by 167
  15. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 159
  16. 2023SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoningcited by 157
  17. 2020Learning from Task Descriptionscited by 78
  18. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 76
  19. 2023Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Modelscited by 66
  20. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
  21. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 42
  22. 2023Phoenix: Democratizing ChatGPT across Languagescited by 32
  23. 2023Improving Contrastive Learning of Sentence Embeddings from AI Feedbackcited by 29
  24. 2022Personalizing Text-to-Image Generation via Aesthetic Gradientscited by 13

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3.

4 papers.

  1. 2024TELLER: A Trustworthy Framework for Explainable, Generalizable and Controllable Fake News Detectioncited by 10
  2. 2024Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMscited by 10
  3. 2024GRATH: Gradual Self-Truthifying for Large Language Modelscited by 6
  4. 2023Distilled Self-Critique of LLMs with Synthetic Data: a Bayesian Perspectivecited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
  2. 2022Critic-Guided Decoding for Controlled Text Generation
  3. 2024PREDILECT: Preferences Delineated with Zero-Shot Language-based Reasoning in Reinforcement Learning
  4. 2026Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation
  5. 2022Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective
  6. 2025Text2Grad: Reinforcement Learning from Natural Language Feedback
  7. 2025Towards Cost-Effective Reward Guided Text Generation
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback