mlsanityData to 14 Oct 2026

Not Eliminate but Aggregate: Post-Hoc Control over Mixture-of-Experts to Address Shortcut Shifts in Natural Language Understanding

arXiv 2406.12060 ↗First posted 17 Jun 2024cs.CL cs.LGcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Recent models for natural language understanding are inclined to exploit simple patterns in datasets, commonly known as shortcuts. These shortcuts hinge on spurious correlations between labels and latent features existing in the training data. At inference time, shortcut-dependent models are likely to generate erroneous predictions under distribution shifts, particularly when some latent features are no longer correlated with the labels. To avoid this, previous studies have trained models to eliminate the reliance on shortcuts. In this study, we explore a different direction: pessimistically aggregating the predictions of a mixture-of-experts, assuming each expert captures relatively different latent features. The experimental results demonstrate that our post-hoc control over the experts significantly enhances the model's robustness to the distribution shift in shortcuts. Besides, we show that our approach has some practical advantages. We also analyze our model and provide results to support the assumption.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 61 of its 86 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

61 of 86 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding30 Jun
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 59

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2019Invariant Risk Minimizationcited by 1,854
  7. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  8. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  9. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  10. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 809
  11. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 742
  12. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
38 more
  1. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  2. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  3. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  4. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 381
  5. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  6. 2020Environment Inference for Invariant Learningcited by 306
  7. 2019Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biasescited by 280
  8. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  9. 2022Improving Out-of-Distribution Robustness via Selective Augmentationcited by 202
  10. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 178
  11. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 175
  12. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 152
  13. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  14. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  15. 2019Towards Debiasing Fact Verification Modelscited by 143
  16. 2019End-to-End Bias Mitigation by Modelling Biases in Corporacited by 128
  17. 2023Change is Hard: A Closer Look at Subpopulation Shiftcited by 120
  18. 2019Unlearn Dataset Bias in Natural Language Inference by Fitting the Residualcited by 119
  19. 2020Towards Debiasing NLU Models from Unknown Biasescited by 116
  20. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  21. 2020Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performancecited by 85
  22. 2020Learning from others' mistakes: Avoiding dataset biases without modeling themcited by 83
  23. 2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78
  24. 2020Explaining The Efficacy of Counterfactually Augmented Datacited by 65
  25. 2020Identifying Spurious Correlations for Robust Text Classificationcited by 59
  26. 2021Causally motivated Shortcut Removal Using Auxiliary Labelscited by 55
  27. 2022Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasetscited by 54
  28. 2019Don't Take the Premise for Granted: Mitigating Artifacts in Natural Language Inferencecited by 49
  29. 2020Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensemblescited by 43
  30. 2021Out-of-distribution Generalization in the Presence of Nuisance-Induced Spurious Correlationscited by 41
  31. 2022Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learningcited by 32
  32. 2021Supervising Model Attention with Human Explanations for Robust Natural Language Inferencecited by 29
  33. 2021End-to-End Self-Debiasing Framework for Robust NLU Trainingcited by 24
  34. 2022Informativeness and Invariance: Two Perspectives on Spurious Correlations in Natural Languagecited by 21
  35. 2022Logical Reasoning with Span-Level Predictions for Interpretable and Robust NLI Modelscited by 18
  36. 2022Nuisances via Negativa: Adjusting for Spurious Correlations via Data Augmentationcited by 18
  37. 2021Uncertainty Calibration for Ensemble-Based Debiasing Methodscited by 16
  38. 2022Debiasing Masks: A New Framework for Shortcut Mitigation in NLUcited by 16

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Investigating Multi-Hop Factual Shortcuts in Knowledge Editing of Large Language Models
  2. 2024A Closer Look into Mixture-of-Experts in Large Language Models
  3. 2025Unveiling Hidden Collaboration within Mixture-of-Experts in Large Language Models
  4. 2025Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say
  5. 2024Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
  6. 2025Mosaic Pruning: A Hierarchical Framework for Generalizable Pruning of Mixture-of-Experts Models
  7. 2024A Survey on Mixture of Experts in Large Language Models
  8. 2026RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing