mlsanityData to 14 Oct 2026

Convex Optimization for Alignment and Preference Learning on a Single GPU

arXiv 2605.23244 ↗First posted 22 May 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Fine-tuning large language models (LLMs) to align with human preferences has driven the success of systems such as Gemini and ChatGPT. However, approaches like Reinforcement Learning from Human Feedback (RLHF) remain computationally expensive and complex. Direct Preference Optimization (DPO) offers a simpler alternative but has limitations such as inconsistent ranking accuracy, high dependence on GPU resources, and expensive hyperparameter tuning. We propose the Convex Optimization for Alignment and Preference Learning Algorithm (COALA): a novel lightweight strategy with strong theoretical guarantees. By leveraging the convex optimization reformulation of neural networks, COALA eliminates the need for a reference model and obtains significant reduction in both training time and VRAM consumption, thus enabling efficient training on a single GPU. Experiments across four datasets--including a 26621-sample synthetic Educational Feedback dataset--and six models (including Llama-3.1-8B) demonstrate COALA's competitive performance and efficiency while utilizing as little as ~17.6% of DPO's total TFLOPs. COALA exhibits stable, monotonically increasing rewards and reaches peak margins in significantly shorter time in comparison to traditional methods such as DPO and ORPO. To the best of our knowledge, this is the first time convex optimization has been effectively applied to preference fine-tuning of LLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 79 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

53 of 79 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets14 Nov
  2. 2025Alignment-Aware Decoding30 Sep
  3. 2025Toward Preference-aligned Large Language Models via Residual-based Model Steering28 Sep
  4. 2024Preference Learning Algorithms Do Not Learn Preference Rankings29 May
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Binary Classifier Optimization for Large Language Model Alignment6 Apr
  7. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  8. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  9. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  11. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
30 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  2. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  3. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  7. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  8. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  9. 2020Evaluation of Text Generation: A Surveycited by 277
  10. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  11. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 118
  12. 2020Neural Networks are Convex Regularizers: Exact Polynomial-time Convex Optimization Formulations for Two-layer Networkscited by 98
  13. 2023Is RLHF More Difficult than Standard RL?cited by 96
  14. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  15. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  16. 2021Acceleration Methodscited by 48
  17. 2022Augmenting Interpretable Models with LLMs during Trainingcited by 38
  18. 2022Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositionscited by 31
  19. 2024Inf-MLLM: Efficient Streaming Inference of Multimodal Large Language Models on a Single GPUcited by 20
  20. 2024A Survey of Sustainability in Large Language Models: Applications, Economics, and Challengescited by 18
  21. 2020Nimble: Lightweight and Parallel GPU Task Scheduling for Deep Learningcited by 17
  22. 2024Path-Consistency with Prefix Enhancement for Efficient Inference in LLMscited by 17
  23. 2025Electricity Demand and Grid Impacts of AI Data Centers: Challenges and Prospectscited by 16
  24. 2022Efficient Global Optimization of Two-Layer ReLU Networks: Quadratic-Time Algorithms and Adversarial Trainingcited by 15
  25. 2024Is One GPU Enough? Pushing Image Generation at Higher-Resolutions with Foundation Modelscited by 14
  26. 2024Propulsion: Steering LLM with Tiny Fine-Tuningcited by 14
  27. 2024Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Timecited by 11
  28. 2024CRONOS: Enhancing Deep Learning with Scalable GPU Accelerated Convex Neural Networkscited by 4
  29. 2018Generalized Symmetric ADMM for Separable Convex Optimizationcited by 3
  30. 2016Globally Optimal Training of Generalized Polynomial Neural Networks with Nonlinear Spectral Methodscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  2. 2024Accelerated Preference Optimization for Large Language Model Alignment
  3. 2024OPTune: Efficient Online Preference Tuning
  4. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  5. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  6. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  7. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization
  8. 2024Towards Efficient Exact Optimization of Language Model Alignment