mlsanityData to 7 Oct 2026

Zephyr: Direct Distillation of LM Alignment

arXiv 2310.16944 ↗First posted 25 Oct 2023cs.LG cs.CLcited by 498

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We aim to produce a smaller language model that is aligned to user intent. Previous research has shown that applying distilled supervised fine-tuning (dSFT) on larger models significantly improves task accuracy; however, these models are unaligned, i.e. they do not respond well to natural prompts. To distill this property, we experiment with the use of preference data from AI Feedback (AIF). Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment. The approach requires only a few hours of training without any additional sampling during fine-tuning. The final result, Zephyr-7B, sets the state-of-the-art on chat benchmarks for 7B parameter models, and requires no human annotation. In particular, results on MT-Bench show that Zephyr-7B surpasses Llama2-Chat-70B, the best open-access RLHF-based model. Code, models, data, and tutorials for the system are available at https://github.com/huggingface/alignment-handbook.

Key facts. Cited by 498 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 28 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

15 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec

Other references · 14

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
2 more
  1. 2023The False Promise of Imitating Proprietary LLMscited by 254
  2. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 22; 2024: 265; 2025: 140; 2026: 71.

498 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  4. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  5. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  6. 2024Large Language Models: A Surveycited by 506
  7. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  9. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  10. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  11. 2025Memory in the Age of AI Agentscited by 285
  12. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
38 more
  1. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  2. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  3. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  4. 2024Generative Representational Instruction Tuningcited by 227
  5. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 205
  6. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  7. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 175
  8. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  9. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  10. 2024Token-level Direct Preference Optimizationcited by 135
  11. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 124
  12. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  13. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  14. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  15. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  16. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  17. 2025Small Models Struggle to Learn from Strong Reasonerscited by 109
  18. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  19. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  20. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  21. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 103
  22. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  23. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  24. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  25. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  26. 2024Noise Contrastive Alignment of Language Models with Explicit Rewardscited by 85
  27. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  28. 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearningcited by 81
  29. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 80
  30. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  31. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 70
  32. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  33. 2024Decoding-time Realignment of Language Modelscited by 67
  34. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 67
  35. 2024Archon: An Architecture Search Framework for Inference-Time Techniquescited by 59
  36. 2025Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learningcited by 58
  37. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  38. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 57

50 most cited of 498 All 498 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment
  2. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  3. 2024Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads
  4. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
  5. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback
  6. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment
  7. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  8. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback