mlsanityData to 7 Oct 2026

Not Everything is All You Need: Toward Low-Redundant Optimization for Large Language Model Alignment

arXiv 2406.12606 ↗First posted 18 Jun 2024cs.CLcited by 7

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models (LLMs) are still struggling in aligning with human preference in complex tasks and scenarios. They are prone to overfit into the unexpected patterns or superficial styles in the training data. We conduct an empirical study that only selects the top-10% most updated parameters in LLMs for alignment training, and see improvements in the convergence process and final performance. It indicates the existence of redundant neurons in LLMs for alignment training. To reduce its influence, we propose a low-redundant alignment method named ALLO, focusing on optimizing the most related neurons with the most useful supervised signals. Concretely, we first identify the neurons that are related to the human preference data by a gradient-based strategy, then identify the alignment-related key tokens by reward models for computing loss. Besides, we also decompose the alignment process into the forgetting and learning stages, where we first forget the tokens with unaligned knowledge and then learn aligned knowledge, by updating different ratios of neurons, respectively. Experimental results on 10 datasets have shown the effectiveness of ALLO. Our code and data are available at urlhttps://github.com/RUCAIBox/ALLO.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 60 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

43 of 60 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  3. 2024Binary Classifier Optimization for Large Language Model Alignment6 Apr
  4. 2024Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective6 Apr
  5. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  6. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  8. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  9. 2022Training language models to follow instructions with human feedback4 Mar
  10. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  10. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
21 more
  1. 2019Machine Unlearningcited by 1,020
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  3. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  4. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  5. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  6. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  7. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  8. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  9. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  10. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  11. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 248
  12. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  13. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  14. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 181
  15. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  16. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  17. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 44
  18. 2024Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraintcited by 39
  19. 2024LoRA-drop: Efficient LoRA Parameter Pruning based on Output Evaluationcited by 38
  20. 2024Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMscited by 33
  21. 2023TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewisecited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 1.

7 papers.

  1. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  2. 2025Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learningcited by 21
  3. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVRcited by 16
  4. 2025L-MTP: Leap Multi-Token Prediction Beyond Adjacent Context for Large Language Modelscited by 14
  5. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMscited by 10
  6. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 2
  7. 2025Explainable reinforcement learning from human feedback to improve alignmentcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  2. 2024Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models
  3. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  4. 2025DiffPO: Diffusion-styled Preference Optimization for Efficient Inference-Time Alignment of Large Language Models
  5. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  6. 2024DeAL: Decoding-time Alignment for Large Language Models
  7. 2025A Survey on Training-free Alignment of Large Language Models
  8. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignment