mlsanityData to 14 Oct 2026

LIONs: An Empirically Optimized Approach to Align Language Models

arXiv 2407.06542 ↗First posted 9 Jul 2024cs.CLcited by 7

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Alignment is a crucial step to enhance the instruction-following and conversational abilities of language models. Despite many recent work proposing new algorithms, datasets, and training pipelines, there is a lack of comprehensive studies measuring the impact of various design choices throughout the whole training process. We first conduct a rigorous analysis over a three-stage training pipeline consisting of supervised fine-tuning, offline preference learning, and online preference learning. We have found that using techniques like sequence packing, loss masking in SFT, increasing the preference dataset size in DPO, and online DPO training can significantly improve the performance of language models. We then train from Gemma-2b-base and LLama-3-8b-base, and find that our best models exceed the performance of the official instruct models tuned with closed-source data and algorithms. Our code and models can be found at urlhttps://github.com/Columbia-NLP-Lab/LionAlignment.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 61 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

41 of 61 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  3. 2024Learn Your Reference Model for Real Good Alignment15 Apr
  4. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  5. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning7 Feb
  6. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  7. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  8. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  9. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  10. 2023LIMA: Less Is More for Alignment18 May
  11. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  12. 2020Learning to summarize from human feedbackcited by 2,588
18 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  2. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  7. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  8. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  9. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  10. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  11. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  12. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  13. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  14. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  15. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  16. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  17. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  18. 2023Teaching Language Models to Self-Improve through Interactive Demonstrationscited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 1.

7 papers.

  1. 2025Think2SQL: Reinforce LLM Reasoning Capabilities for Text2SQLcited by 14
  2. 2025Aligning Large Language Models with Implicit Preferences from User-Generated Contentcited by 9
  3. 2024Data-Centric Improvements for Enhancing Multi-Modal Understanding in Spoken Conversation Modelingcited by 3
  4. 2025FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankingscited by 3
  5. 2025NAACL2025 Tutorial: Adaptation of Large Language Modelscited by 2
  6. 2025GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Datacited by 1
  7. 2026Direct Hidden-State Alignment: Mapping and Controlling Preference Expression in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
  2. 2024Aligner: Efficient Alignment by Learning to Correct
  3. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  4. 2025Towards Efficient and Effective Alignment of Large Language Models
  5. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  6. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
  7. 2024Is Free Self-Alignment Possible?
  8. 2024Hybrid Alignment Training for Large Language Models