mlsanityData to 14 Oct 2026

The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translation

arXiv 2310.20620 ↗First posted 31 Oct 2023cs.CL cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Continuous-output neural machine translation (CoNMT) replaces the discrete next-word prediction problem with an embedding prediction. The semantic structure of the target embedding space (i.e., closeness of related words) is intuitively believed to be crucial. We challenge this assumption and show that completely random output embeddings can outperform laboriously pretrained ones, especially on larger datasets. Further investigation shows this surprising effect is strongest for rare words, due to the geometry of their embeddings. We shed further light on this finding by designing a mixed strategy that combines random and pre-trained embeddings for different tokens.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

22 of 30 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  7. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  10. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  11. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  12. 2020Language-agnostic BERT Sentence Embeddingcited by 627
10 more
  1. 2020Nearest Neighbor Machine Translationcited by 209
  2. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  3. 2021Adaptive Semiparametric Language Modelscited by 83
  4. 2014Improving zero-shot learning by mitigating the hubness problemcited by 54
  5. 2022Problems with Cosine as a Measure of Embedding Similarity for High Frequency Wordscited by 49
  6. 2019compare-mt: A Tool for Holistic Comparison of Language Generation Systemscited by 43
  7. 2021von Mises-Fisher Loss: An Exploration of Embedding Geometries for Supervised Learningcited by 42
  8. 2018Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputscited by 27
  9. 2020Contextual Embeddings: When Are They Worth It?cited by 15
  10. 2022Low-Rank Softmax Can Have Unargmaxable Classes in Theory but Rarely in Practicecited by 15

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 1.

2 papers.

  1. 2023GIVT: Generative Infinite-Vocabulary Transformerscited by 80
  2. 2025Keep your distance: learning dispersed embeddings on mathbbSmcited by 1