mlsanityData to 14 Oct 2026

Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data

arXiv 2504.09895 ↗First posted 14 Apr 2025cs.CL cs.AI cs.LGcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models~(LLMs) are expected to be helpful, harmless, and honest. In different alignment scenarios, such as safety, confidence, and general preference alignment, binary preference data collection and reward modeling are resource-intensive but play a central role in transferring human preferences. In this work, we explore using the similarity between sampled generations and reference answers as a supplementary reward function for alignment. When unary reference answers are available, such similarity-based rewards can circumvent the need for binary preference data and explicit reward modeling. We introduce textitRefAlign, a versatile REINFORCE-style alignment algorithm that does not rely on reward or reference models. RefAlign utilizes language generation evaluation metrics, such as BERTScore, between sampled generations and reference answers as surrogate rewards. Beyond general preference optimization, RefAlign can be naturally extended to diverse scenarios, including safety and confidence alignment, by combining similarity-based rewards with task-specific objectives. Across multiple scenarios, RefAlign achieves performance comparable to prior alignment methods while operating without binary preference data or reward models. The code is available at https://github.com/mzhaoshuai/RefAlign.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 67 of its 89 references that are in the map. First posted on arXiv 14 Apr 2025.

Builds on

67 of 89 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  3. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  4. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  5. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  6. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  7. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  8. 2023Alignment for Honesty12 Dec
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  10. 2023Preference-grounded Token-level Guidance for Language Model Fine-tuning1 Jun
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback24 May
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
1 more
  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 54

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  12. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
38 more
  1. 2017On Calibration of Modern Neural Networkscited by 4,960
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2023Mistral 7Bcited by 3,164
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  13. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  14. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  15. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  16. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  17. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  18. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  19. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  20. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  21. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  22. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  23. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  24. 2016Self-critical Sequence Training for Image Captioningcited by 547
  25. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  26. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  27. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  28. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  29. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  30. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  31. 2024Iterative Reasoning Preference Optimizationcited by 246
  32. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  33. 2023Fine-tuning Language Models for Factualitycited by 221
  34. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  35. 2023R-Tuning: Instructing Large Language Models to Say `I Don't Know'cited by 111
  36. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 105
  37. 2023Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Modelscited by 91
  38. 2023On the Calibration of Large Language Models and Alignmentcited by 73

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  2. 2025Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimizationcited by 5
  3. 2026References Improve LLM Alignment in Non-Verifiable Domainscited by 1
  4. 2026VIA-SD: Verification via Intra-Model Routing for Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023On Diversified Preferences of Large Language Model Alignment
  2. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  3. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment
  4. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  5. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  6. 2025Energy-Based Reward Models for Robust Language Model Alignment
  7. 2025ComPO: Preference Alignment via Comparison Oracles
  8. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques