mlsanityData to 7 Oct 2026

Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safety

arXiv 2509.12936 ↗First posted 16 Sep 2025cs.LG cs.CLcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Large language models (LLMs) require careful alignment to balance competing objectives - factuality, safety, conciseness, proactivity, and diversity. Existing studies focus on individual techniques or specific dimensions, lacking a holistic assessment of the inherent trade-offs. We propose a unified evaluation framework that compares LLM alignment methods (PPO, DPO, ORPO, KTO) across these five axes, using both in-distribution and out-of-distribution datasets. Leveraging a specialized LLM-as-Judge prompt, validated through human studies, we reveal that DPO and KTO excel in factual accuracy, PPO and DPO lead in safety, and PPO best balances conciseness with proactivity. Our findings provide insights into trade-offs of common alignment methods, guiding the development of more balanced and reliable LLMs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 34 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

18 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  7. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  8. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  9. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 92
  10. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  11. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 57
  12. 2024Benchmarking Language Model Creativity: A Case Study on Code Generationcited by 38
5 more
  1. 2024Creativity Has Left the Chat: The Price of Debiasing Language Modelscited by 29
  2. 2024Exploring Precision and Recall to assess the quality and diversity of LLMscited by 27
  3. 2024Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methodscited by 26
  4. 2024CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraintscited by 22
  5. 2024Look Before You Leap: Enhancing Attention and Vigilance Regarding Harmful Content with GuidelineLLMcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustnesscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  2. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  3. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples
  4. 2025On Evaluating LLM Alignment by Evaluating LLMs as Judges
  5. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences
  6. 2025The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual context
  7. 2023Large Language Model Alignment: A Survey
  8. 2026Are Aligned Large Language Models Still Misaligned?