mlsanityData to 7 Oct 2026

Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback

arXiv 2401.11458 ↗First posted 21 Jan 2024cs.CLcited by 25

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

The success of AI assistants based on Language Models (LLMs) hinges on Reinforcement Learning from Human Feedback (RLHF) to comprehend and align with user intentions. However, traditional alignment algorithms, such as PPO, are hampered by complex annotation and training requirements. This reliance limits the applicability of RLHF and hinders the development of professional assistants tailored to diverse human preferences. In this work, we introduce textitLinear Alignment, a novel algorithm that aligns language models with human preferences in one single inference step, eliminating the reliance on data annotation and model training. Linear alignment incorporates a new parameterization for policy optimization under divergence constraints, which enables the extraction of optimal policy in a closed-form manner and facilitates the direct estimation of the aligned response. Extensive experiments on both general and personalized preference datasets demonstrate that linear alignment significantly enhances the performance and efficiency of LLM alignment across diverse scenarios. Our code and dataset is published on urlhttps://github.com/Wizardcoast/LinearAlignment.git.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 53 references that are in the map. First posted on arXiv 21 Jan 2024.

Builds on

43 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  2. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment7 Nov
  3. 2023RAIN: Your Language Models Can Align Themselves without Finetuning13 Sep
  4. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2015Trust Region Policy Optimizationcited by 2,503
  12. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
26 more
  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  6. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  9. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  10. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  11. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  12. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  13. 2022Teaching language models to support answers with verified quotescited by 290
  14. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  15. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  16. 2023Controlled Decoding from Language Modelscited by 133
  17. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  18. 2019Better Exploration with Optimistic Actor-Criticcited by 88
  19. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  20. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 42
  21. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  22. 2023Specific versus General Principles for Constitutional AIcited by 41
  23. 2023The Benefits of Bad Advice: Autocontrastive Decoding across Model Layerscited by 28
  24. 2020Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learningcited by 21
  25. 2024Parameter-Efficient Detoxification with Contrastive Decodingcited by 18
  26. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operatorscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 12; 2026: 6.

25 papers.

Sort:

By area:

  1. 2024Navigating the OverKill in Large Language Modelscited by 40
  2. 2025A Survey of Direct Preference Optimizationcited by 35
  3. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  4. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  5. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Modelscited by 25
  6. 2025CoSteer: Collaborative Decoding-Time Personalization via Local Delta Steeringcited by 12
  7. 2025LLM-Auction: Generative Auction towards LLM-Native Advertisingcited by 10
  8. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
  9. 2025T-POP: Test-Time Personalization with Online Preference Feedbackcited by 6
  10. 2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
  11. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Timecited by 5
  12. 2025How Well Can Preference Optimization Generalize Under Noisy Feedback?cited by 3
13 more
  1. 2026EXACT: Explicit Attribute-Guided Decoding-Time Personalizationcited by 3
  2. 2025A Survey on Training-free Alignment of Large Language Modelscited by 2
  3. 2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignmentcited by 2
  4. 2026Enhancing Persona Following at Decoding Time via Dynamic Importance Estimation for Role-Playing Agentscited by 2
  5. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1
  6. 2024What's New in My Data? Novelty Exploration via Contrastive Generationcited by 1
  7. 2025Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?cited by 1
  8. 2025MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selectioncited by 0
  9. 2025Factors That Support Grounded Responses in LLM Conversations: A Rapid Reviewcited by 0
  10. 2026Synthesizing Instruction-Tuning Datasets with Contrastive Decodingcited by 0
  11. 2026Training-Free Refusal of MCP Exploits via Retrieval-Augmented Generationcited by 0
  12. 2026MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Modelscited by 0
  13. 2026REAR: Test-time Preference Realignment through Reward Decompositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  2. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  3. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
  4. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  5. 2023Preference Ranking Optimization for Human Alignment
  6. 2024Understanding the Learning Dynamics of Alignment with Human Feedback
  7. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
  8. 2024Towards Efficient Exact Optimization of Language Model Alignment