# Knowledge Injection Exists in MoE? Exploring Expert-Aware Contrast Decoding in MoE for Mitigating LLMs'Hallucinations

arXiv: https://arxiv.org/abs/2607.20426 · HTML: https://mlsanity.com/paper/2607.20426/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in MoE large language models, one of 12 directions in Mixture-of-experts.
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 8 May 2026.

## Placed in

- Mixture-of-experts / [MoE large language models](https://mlsanity.com/moe/moe-large-language-models/)

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://mlsanity.com/paper/2305.14739/) (2023)
- [Can We Edit Factual Knowledge by In-Context Learning?](https://mlsanity.com/paper/2305.12740/) (2023)
- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training](https://mlsanity.com/paper/2406.16554/) (2024)
- [Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts](https://mlsanity.com/paper/2405.11273/) (2024)
- [Generating Benchmarks for Factuality Evaluation of Language Models](https://mlsanity.com/paper/2307.06908/) (2023)
- [Towards Mitigating Hallucination in Large Language Models via Self-Reflection](https://mlsanity.com/paper/2310.06271/) (2023)
- [Alleviating Hallucinations of Large Language Models through Induced Hallucinations](https://mlsanity.com/paper/2312.15710/) (2023)
- [KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning](https://mlsanity.com/paper/2401.12863/) (2024)
- [A Closer Look into Mixture-of-Experts in Large Language Models](https://mlsanity.com/paper/2406.18219/) (2024)
- [The Benefits of Bad Advice: Autocontrastive Decoding across Model Layers](https://mlsanity.com/paper/2305.01628/) (2023)
- [XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection](https://mlsanity.com/paper/2403.18926/) (2024)
- [HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts](https://mlsanity.com/paper/2402.12656/) (2024)
- [Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast](https://mlsanity.com/paper/2405.14507/) (2024)
- [Improving Factuality in Large Language Models via Decoding-Time Hallucinatory and Truthful Comparators](https://mlsanity.com/paper/2408.12325/) (2024)
- [A Frustratingly Simple Decoding Method for Neural Text Generation](https://mlsanity.com/paper/2305.12675/) (2023)
- [Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion](https://mlsanity.com/paper/2310.04361/) (2023)
- [Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling](https://mlsanity.com/paper/2409.11283/) (2024)
- [Inv-Entropy: A Fully Probabilistic Framework for Uncertainty Quantification in Language Models](https://mlsanity.com/paper/2506.09684/) (2025)
- [CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts](https://mlsanity.com/paper/2410.16077/) (2024)

## Built on by (most cited first, 2 of 2)

- [The Safeguard Worked. Is the LLM System Safer?](https://mlsanity.com/paper/2609.00519/) (2026)
- [Only Pay What You Must Spend: On-Demand Privacy Budget Payment for Differentially Private RAG](https://mlsanity.com/paper/2609.27406/) (2026)

## Abstract (arXiv)

Existing LLM hallucination mitigation methods, including prompt engineering and model optimization, either hardly alter models'internal knowledge or have poor cross-domain generalization. Contrastive decoding mitigates hallucinations by using layer-wise differences in LLMs. However, prior studies only explore transformer-based models (e.g., GPT), ignoring other effective frameworks like mixture-of-experts (MoE) models. Since MoE alters the traditional transformer architecture, we conduct empirical studies to investigate whether similar layer-wise differences exist in MoEs. Our results show that they do not exist in MoE with shared experts; nevertheless, across different MoEs, higher layers exhibit distinct expert activation patterns between factual and non-factual outputs. Building on these, we propose EAACD, an expert-aware adaptive contrast decoding that uses expert differences in MoE's higher layers to mitigate hallucinations on QA tasks. EAACD splits high-layer experts into a higher-reliability group and several lower-reliability groups based on their confidence and consistency. It contrasts the higher-reliability group's prediction with each lower-reliability group's prediction to calibrate the model's original predictions. To strengthen this contrast, EAACD amplifies hallucinations from lower-reliability experts via attention and masking to provide stronger negative references. EAACD outperforms all baselines on four datasets.