mlsanityData to 7 Oct 2026

Reasons to Reject? Aligning Language Models with Judgments

arXiv 2312.14591 ↗First posted 22 Dec 2023cs.CLcited by 15

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

As humans, we consistently interact with our peers and receive feedback in the form of natural language. This language feedback allows us to maintain appropriate behavior, and rectify potential errors. The question arises naturally: can we use language feedback to align large language models (LLMs)? In contrast to previous research that aligns LLMs with scalar rewards, we present the first systematic exploration of alignment through the lens of language feedback (i.e., judgment). We start with an in-depth investigation of potential methods that can be adapted for aligning LLMs with judgments, revealing that these methods cannot fully capitalize on judgments. To facilitate more effective utilization of judgments, we propose a novel framework, Contrastive Unlikelihood Training (CUT), that allows for fine-grained inappropriate content detection and correction based on judgments. Our results show that, with merely 1317 off-the-shelf judgment data, CUT (LLaMA2-13b) can beat the 175B DaVinci003 and surpass the best baseline by 50.84 points on AlpacaEval. CUT (LLaMA2-chat-13b) can also align LLMs in an iterative fashion using up-to-date model-specific judgments, improving performance from 81.09 to 91.68 points on AlpacaEval. Further analysis suggests that judgments hold greater potential than rewards in LLM alignment.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 54 of its 69 references that are in the map. First posted on arXiv 22 Dec 2023.

Builds on

54 of 69 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Let's Verify Step by Step31 May
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  4. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  11. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
37 more
  1. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  2. 2023LIMA: Less Is More for Alignmentcited by 1,080
  3. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  4. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  6. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  7. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  8. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  9. 2022Self-critiquing models for assisting human evaluatorscited by 372
  10. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  11. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  12. 2022Generating Sequences by Learning to Self-Correctcited by 271
  13. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  14. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  15. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  16. 2023Generative Judge for Evaluating Alignmentcited by 163
  17. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  18. 2023Training Language Models with Language Feedback at Scalecited by 125
  19. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  20. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 104
  21. 2023Shepherd: A Critic for Language Model Generationcited by 96
  22. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59
  23. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  24. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 42
  25. 2022On Improving Summarization Factual Consistency from Natural Language Feedbackcited by 38
  26. 2021Learning to Repair: Repairing model output errors after deployment using a dynamic memory of feedbackcited by 35
  27. 2022Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedbackcited by 35
  28. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 34
  29. 2022Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deploymentcited by 25
  30. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23
  31. 2016Dialog-based Language Learningcited by 21
  32. 2023Stabilizing RLHF through Advantage Model and Selective Rehearsalcited by 14
  33. 2023MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Modelscited by 13
  34. 2023Constructive Large Language Models Alignment with Diverse Feedbackcited by 9
  35. 2022PeerDA: Data Augmentation via Modeling Peer Relation for Span Identification Taskscited by 7
  36. 2022From Cloze to Comprehension: Retrofitting Pre-trained Masked Language Model to Pre-trained Machine Readercited by 2
  37. 2023mPMR: A Multilingual Pre-trained Machine Reader at Scalecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 1; 2026: 4.

15 papers.

Sort:

By area:

  1. 2024Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agentscited by 94
  2. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  3. 2024RaFe: Ranking Feedback Improves Query Rewriting for RAGcited by 47
  4. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  5. 2024UICrit: Enhancing Automated Design Evaluation with a UICritique Datasetcited by 21
  6. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  7. 2024MACAROON: Training Vision-Language Models To Be Your Engaged Partnerscited by 10
  8. 2024ICDPO: Effectively Borrowing Alignment Capability of Others via In-context Direct Preference Optimizationcited by 6
  9. 2024EventRL: Enhancing Event Extraction with Outcome Supervision for Large Language Modelscited by 3
  10. 2024FaGeL: Fabric LLMs Agent empowered Embodied Intelligence Evolution with Autonomous Human-Machine Collaborationcited by 2
  11. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  12. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1
3 more
  1. 2026Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimizationcited by 1
  2. 2024Natural Language Fine-Tuningcited by 0
  3. 2026Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Aligning Black-box Language Models with Human Judgments
  2. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
  3. 2025DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decomposition
  4. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
  5. 2024Improving the Robustness of Large Language Models via Consistency Alignment
  6. 2023Aligning Language Models with Offline Learning from Human Feedback
  7. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments
  8. 2023Aligning Large Language Models through Synthetic Feedback