mlsanityData to 7 Oct 2026

Adding Alignment Control to Language Models

arXiv 2503.04346 ↗First posted 6 Mar 2025cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Post-training alignment has increasingly become a crucial factor in enhancing the usability of language models (LMs). However, the strength of alignment varies depending on individual preferences. This paper proposes a method to incorporate alignment control into a single model, referred to as CLM. This approach adds one identity layer preceding the initial layers and performs preference learning only on this layer to map unaligned input token embeddings into the aligned space. Experimental results demonstrate that this efficient fine-tuning method performs comparable to full fine-tuning. During inference, the input embeddings are processed through the aligned and unaligned layers, which are then merged through the interpolation coefficient. By controlling this parameter, the alignment exhibits a clear interpolation and extrapolation phenomenon.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 39 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

33 of 39 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model24 Oct
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024Model Extrapolation Expedites Alignment25 Apr
  4. 2024Decoding-time Realignment of Language Models5 Feb
  5. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2023Let's Verify Step by Stepcited by 3,598
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
15 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  4. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  7. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  8. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  9. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  10. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  11. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  12. 2019A Theory of Regularized Markov Decision Processescited by 279
  13. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  14. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 99
  15. 2020On the Transformer Growth for Progressive BERT Trainingcited by 49

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning Large Language Models with Representation Editing: A Control Perspective
  2. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning
  3. 2024Understanding Layer Significance in LLM Alignment
  4. 2025A Survey on Training-free Alignment of Large Language Models
  5. 2024Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignment
  6. 2024Aligner: Efficient Alignment by Learning to Correct
  7. 2024Aligners: Decoupling LLMs and Alignment
  8. 2023ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference