mlsanityData to 14 Oct 2026

Flexible Realignment of Language Models

arXiv 2506.12704 ↗First posted 15 Jun 2025cs.CL cs.AIcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Realignment becomes necessary when a language model (LM) fails to meet expected performance. We propose a flexible realignment framework that supports quantitative control of alignment degree during training and inference. This framework incorporates Training-time Realignment (TrRa), which efficiently realigns the reference model by leveraging the controllable fusion of logits from both the reference and already aligned models. For example, TrRa reduces token usage by 54.63% on DeepSeek-R1-Distill-Qwen-1.5B without any performance degradation, outperforming DeepScaleR-1.5B's 33.86%. To complement TrRa during inference, we introduce a layer adapter that enables smooth Inference-time Realignment (InRa). This adapter is initialized to perform an identity transformation at the bottom layer and is inserted preceding the original layers. During inference, input embeddings are simultaneously processed by the adapter and the original layer, followed by the remaining layers, and then controllably interpolated at the logit level. We upgraded DeepSeek-R1-Distill-Qwen-7B from a slow-thinking model to one that supports both fast and slow thinking, allowing flexible alignment control even during inference. By encouraging deeper reasoning, it even surpassed its original performance.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 31 of its 41 references that are in the map. First posted on arXiv 15 Jun 2025.

Builds on

31 of 41 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  11. 2017Deep reinforcement learning from human preferencescited by 3,912
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
19 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2024Qwen2 Technical Reportcited by 2,120
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  11. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  12. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  13. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  14. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  15. 2019A Theory of Regularized Markov Decision Processescited by 286
  16. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  17. 2024Decoding-time Realignment of Language Modelscited by 67
  18. 2020On the Transformer Growth for Progressive BERT Trainingcited by 49
  19. 2024Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Modelcited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Researchcited by 19
  2. 2025Proximal Supervised Fine-Tuningcited by 16
  3. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2024Aligning Large Language Models with Representation Editing: A Control Perspective
  2. 2026AlignTune: Modular Toolkit for Post-Training Alignment of Large Language Models
  3. 2024InfAlign: Inference-aware language model alignment
  4. 2025AlignFreeze: Navigating the Impact of Realignment on the Layers of Multilingual Models Across Diverse Languages
  5. 2025The Realignment Problem: When Right becomes Wrong in LLMs
  6. 2024Aligner: Efficient Alignment by Learning to Correct
  7. 2023Aligner: One Global Token is Worth Millions of Parameters When Aligning Large Language Models
  8. 2025Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models