mlsanityData to 7 Oct 2026

H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs

arXiv 2411.17792 ↗First posted 26 Nov 2024cs.CL cs.AI cs.LGcited by 11

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

The alignment of pre-trained LLMs continues to draw significant attention from both industry and academia, aiming to ensure responses that are helpful, harmless, and honest. However, identifying a point in the model's representation subspace that simultaneously satisfies all these properties remains challenging. H3Fusion addresses this challenge by introducing a mixture-of-experts (MoE)-based fusion mechanism that models alignment as a controllable drift within the subspace, guided by a drift-regularization loss to balance competing alignment dimensions. Furthermore, we formulate the alignment by finding a dual objective of harnessing the distance of generated embeddings and alignment embeddings, and introduce a gating loss by canalizing the activations on the contributing experts. Extensive evaluations of three benchmark datasets show that H3Fusion is more helpful, less harmful, and more honest in three aspects: it outperforms each individually aligned model by 11.37%, and provides stronger robustness compared to the state-of-the-art LLM ensemble approaches by 13.77% and model-merging approaches by 6.18%. Code is available at https://github.com/git-disl/h3fusion.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 48 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

34 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Large Language Model Alignment: A Survey26 Sep
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  10. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  11. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
19 more
  1. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  2. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  3. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  5. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  6. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  8. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  9. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  10. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  11. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  12. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  13. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  14. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  15. 2024TruthX: Alleviating Hallucinations by Editing Large Language Models in Truthful Spacecited by 90
  16. 2023Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Modelscited by 65
  17. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
  18. 2024Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmeticcited by 50
  19. 2024LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversitycited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 120
  2. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  3. 2025Too Helpful, Too Harmless, Too Honest or Just Right?cited by 9
  4. 2025We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrongcited by 3
  5. 2025Dynamic Optimizations of LLM Ensembles with Two-Stage Reinforcement Learning Agentscited by 2
  6. 2026When the Model Said 'No Comment', We Knew Helpfulness Was Dead, Honesty Was Alive, and Safety Was Terrifiedcited by 1
  7. 2026From Parameters to Behaviors: A Survey of Model Fusion for Large Language Modelscited by 1
  8. 2025Icon2: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulationcited by 0
  9. 2026Can Large Language Models Make Everyone Happy?cited by 0
  10. 2026The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imagingcited by 0
  11. 2026Vision Verification Enhanced Fusion of VLMs for Efficient Visual Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Combining Domain and Alignment Vectors to Achieve Better Knowledge-Safety Trade-offs in LLMs
  2. 2025PIKA: Expert-Level Synthetic Datasets for Post-Training Alignment from Scratch
  3. 2024Aligners: Decoupling LLMs and Alignment
  4. 2024UNA: A Unified Supervised Framework for Efficient LLM Alignment Across Feedback Types
  5. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment
  6. 2024REAL: Response Embedding-based Alignment for LLMs
  7. 2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning
  8. 2024Efficient Alignment of Large Language Models via Data Sampling