References Improve LLM Alignment in Non-Verifiable Domains
On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning
Abstract (arXiv)
While Reinforcement Learning with Verifiable Rewards (RLVR) has shown strong effectiveness in reasoning tasks, it cannot be directly applied to non-verifiable domains lacking ground-truth verifiers, such as LLM alignment. In this work, we investigate whether reference-guided LLM-evaluators can bridge this gap by serving as soft "verifiers". First, we design evaluation protocols that enhance LLM-based evaluators for LLM alignment using reference outputs. Through comprehensive experiments, we show that a reference-guided approach substantially improves the accuracy of less capable LLM-judges using references from frontier models; stronger LLM-judges can also be enhanced by high-quality (i.e., human-written) references. Building on these improved judges, we demonstrate the utility of high-quality references in alignment tuning, where LLMs guided with references are used as judges to self-improve. We show that reference-guided self-improvement yields clear gains over both direct SFT on reference outputs and self-improvement with reference-free judges, achieving performance comparable to training with ArmoRM, a strong finetuned reward model. Specifically, our method achieves 73.1% and 58.7% on AlpacaEval and Arena-Hard with Llama-3-8B-Instruct, and 70.0% and 74.1% with Qwen2.5-7B, corresponding to average absolute gains of +20.2 / +17.1 points over SFT distillation and +5.3 / +3.6 points over reference-free self-improvement on AlpacaEval / Arena-Hard. These results highlight the potential of using reference-guided LLM-evaluators to enable effective LLM post-training in non-verifiable domains.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 56 references that are in the map. First posted on arXiv 18 Feb 2026.
Builds on
40 of 56 references.
Closest ancestors · 1
Same direction, newest first
Other references · 39
- 2023GPT-4 Technical Reportcited by 18,132
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2024GPT-4o System Cardcited by 4,467
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2023Qwen Technical Reportcited by 3,273
- 2023Mistral 7Bcited by 3,065
- 2020Learning to summarize from human feedbackcited by 2,562
27 more
- 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
- 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
- 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
- 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
- 2024Self-Rewarding Language Modelscited by 654
- 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
- 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
- 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
- 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
- 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
- 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
- 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
- 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
- 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
- 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
- 2024Generative Reward Modelscited by 148
- 2025Reinforcement Learning with Rubric Anchorscited by 108
- 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
- 2024How to Evaluate Reward Models for RLHFcited by 84
- 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
- 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
- 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
- 2024Self-rationalization improves LLM as a fine-grained judgecited by 25
- 2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 19
- 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction followingcited by 17
- 2024HREF: Human Response-Guided Evaluation of Instruction Following in Language Modelscited by 8
- 2025Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Datacited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training
- 2025LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignment
- 2025RLPR: Extrapolating RLVR to General Domains without Verifiers
- 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
- 2026Enhancing LLM Metacognition via Cognitive Pairwise Training
- 2026Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks
- 2026Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning
- 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates