mlsanityData to 14 Oct 2026

Xwin-LM: Strong and Scalable Alignment Practice for LLMs

arXiv 2405.20335 ↗First posted 30 May 2024cs.CLcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

In this work, we present Xwin-LM, a comprehensive suite of alignment methodologies for large language models (LLMs). This suite encompasses several key techniques, including supervised finetuning (SFT), reward modeling (RM), rejection sampling finetuning (RS), and direct preference optimization (DPO). The key components are as follows: (1) Xwin-LM-SFT, models initially finetuned with high-quality instruction data; (2) Xwin-Pair, a large-scale, multi-turn preference dataset meticulously annotated using GPT-4; (3) Xwin-RM, reward models trained on Xwin-Pair, developed at scales of 7B, 13B, and 70B parameters; (4) Xwin-Set, a multiwise preference dataset in which each prompt is linked to 64 unique responses generated by Xwin-LM-SFT and scored by Xwin-RM; (5) Xwin-LM-RS, models finetuned with the highest-scoring responses from Xwin-Set; (6) Xwin-LM-DPO, models further optimized on Xwin-Set using the DPO algorithm. Our evaluations on AlpacaEval and MT-bench demonstrate consistent and significant improvements across the pipeline, demonstrating the strength and scalability of Xwin-LM. The repository https://github.com/Xwin-LM/Xwin-LM will be continually updated to foster community research.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 11 of its 20 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

11 of 20 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 10

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  9. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  10. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  2. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  3. 2024Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arenacited by 26

Similar, not cited

Close in wording, no citation either way

  1. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  2. 2025ComPO: Preference Alignment via Comparison Oracles
  3. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment
  4. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences
  5. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  6. 2025CM-Align: Consistency-based Multilingual Alignment for Large Language Models
  7. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  8. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques