mlsanityData to 7 Oct 2026

Mixture Models for Diverse Machine Translation: Tricks of the Trade

arXiv 1902.07816 ↗First posted 20 Feb 2019cs.CL cs.LGcited by 77

Not on the map. Why?

Abstract (arXiv)

Mixture models trained via EM are among the simplest, most widely used and well understood latent variable models in the machine learning literature. Surprisingly, these models have been hardly explored in text generation applications such as machine translation. In principle, they provide a latent variable to control generation and produce a diverse set of hypotheses. In practice, however, mixture models are prone to degeneracies---often only one component gets trained or the latent variable is simply ignored. We find that disabling dropout noise in responsibility computation is critical to successful training. In addition, the design choices of parameterization, prior distribution, hard versus soft EM and online versus offline assignment can dramatically affect model performance. We develop an evaluation protocol to assess both quality and diversity of generations against multiple references, and provide an extensive empirical study of several mixture model variants. Our analysis shows that certain types of mixture models are more robust and offer the best trade-off between translation quality and diversity compared to variational models and diverse decoding approaches.footnoteCode to reproduce the results in this paper is available at urlhttps://github.com/pytorch/fairseq

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 19; 2022: 15; 2023: 14; 2024: 16; 2025: 9; 2026: 3.

77 papers.

Sort:

By area:

  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 299
  3. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  4. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  5. 2021Supporting Clustering with Contrastive Learningcited by 90
  6. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 73
  7. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 65
  8. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  9. 2021Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitivescited by 50
  10. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  11. 2022Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 44
  12. 2021Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policycited by 32
38 more
  1. 2024GigaHands: A Massive Annotated Dataset of Bimanual Hand Activitiescited by 32
  2. 2022A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generationcited by 28
  3. 2023Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizercited by 28
  4. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  5. 2023Cousins Of The Vendi Score: A Family Of Similarity-Based Diversity Metrics For Science And Machine Learningcited by 24
  6. 2021Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMTcited by 21
  7. 2021Data Expansion using Back Translation and Paraphrasing for Hate Speech Detectioncited by 20
  8. 2022One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translationcited by 20
  9. 2022Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Expertscited by 19
  10. 2024Improving Diversity of Commonsense Generation by Large Language Models via In-Context Learningcited by 19
  11. 2021Graph Classification by Mixture of Diverse Expertscited by 16
  12. 2024QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translationcited by 16
  13. 2021multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoningcited by 15
  14. 2023Learning Topology-Specific Experts for Molecular Property Predictioncited by 15
  15. 2021CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Imagescited by 14
  16. 2023Universal Self-Adaptive Promptingcited by 14
  17. 2022Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priorscited by 13
  18. 2021How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation?cited by 12
  19. 2022An Equal-Size Hard EM Algorithm for Diverse Dialogue Generationcited by 11
  20. 2024Generating Diverse and High-Quality Texts by Minimum Bayes Risk Decodingcited by 11
  21. 2024Mix of Experts Language Model for Named Entity Recognitioncited by 11
  22. 2021Diverse and Specific Clarification Question Generation with Keywordscited by 10
  23. 2021Mixup Decoding for Diverse Machine Translationcited by 10
  24. 2022DeepGen: Diverse Search Ad Generation and Real-Time Customizationcited by 10
  25. 2023Adversarial Clean Label Backdoor Attacks and Defenses on Text Classification Systemscited by 10
  26. 2023Natural Language Generation for Advertising: A Surveycited by 10
  27. 2023Diversify Question Generation with Retrieval-Augmented Style Transfercited by 10
  28. 2024To Diverge or Not to Diverge: A Morphosyntactic Perspective on Machine Translation vs Human Translationcited by 10
  29. 2024AmbigNLG: Addressing Task Ambiguity in Instruction for NLGcited by 10
  30. 2024Conditional computation in neural networks: principles and research trendscited by 10
  31. 2025Uncertainty-Aware Decoding with Minimum Bayes Riskcited by 10
  32. 2025Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Abilitycited by 10
  33. 2025Reference-free Evaluation Metrics for Text Generation: A Surveycited by 9
  34. 2021Mixed Cross Entropy Loss for Neural Machine Translationcited by 8
  35. 2021WeTS: A Benchmark for Translation Suggestioncited by 8
  36. 2021Simple or Complex? Complexity-Controllable Question Generation with Soft Templates and Deep Mixture of Experts Modelcited by 8
  37. 2024Sentiment-oriented Transformer-based Variational Autoencoder Network for Live Video Commentingcited by 7
  38. 2022Best-k Search Algorithm for Neural Text Generationcited by 6

50 most cited of 77 All 77 (JSON)