mlsanityData to 14 Oct 2026

L3Ms -- Lagrange Large Language Models

arXiv 2410.21533 ↗First posted 28 Oct 2024cs.LG cs.AI cs.CLcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Supervised fine-tuning (SFT) and alignment of large language models (LLMs) are key steps in providing a good user experience. However, the concept of an appropriate alignment is inherently application-dependent, and current methods often rely on heuristic choices to drive optimization. In this work, we formulate SFT and alignment as a constrained optimization problem: the LLM is fine-tuned on a task while being required to meet application-specific requirements, without resorting to heuristics. To solve this, we propose Lagrange Large Language Models (L3Ms), which employ logarithmic barriers to enforce the constraints. This approach allows for the customization of L3Ms across diverse applications while avoiding heuristic-driven processes. We experimentally demonstrate the versatility and efficacy of L3Ms in achieving tailored alignments for various applications.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 41 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

26 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment29 Feb
  2. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  6. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  10. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  11. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  12. 2021Unified Pre-training for Program Understanding and Generationcited by 366
11 more
  1. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 341
  2. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  3. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  4. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  5. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  6. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  7. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 90
  8. 2019Constrained Deep Networks: Lagrangian Optimization via Log-Barrier Extensionscited by 30
  9. 2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 25
  10. 2015Relaxed Logarithmic Barrier Function Based Model Predictive Control of Linear Systemscited by 8
  11. 2023A Stochastic-Gradient-based Interior-Point Algorithm for Solving Smooth Bound-Constrained Optimization Problemscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  2. 2025E-Scores for (In)Correctness Assessment of Generative Model Outputscited by 2
  3. 2026Provable Last-Iterate Convergence for Multi-Objective Safe LLM Alignment via Optimistic Primal-Dualcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment of large language models with constrained learning
  2. 2023Aligner: One Global Token is Worth Millions of Parameters When Aligning Large Language Models
  3. 2024Efficient Alignment of Large Language Models via Data Sampling
  4. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
  5. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  6. 2026AlignTune: Modular Toolkit for Post-Training Alignment of Large Language Models
  7. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  8. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges