# Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs

arXiv: https://arxiv.org/abs/1812.04616 · HTML: https://mlsanity.com/paper/1812.04616/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2018.

## Built on by (most cited first, 27 of 27)

- [GIVT: Generative Infinite-Vocabulary Transformers](https://mlsanity.com/paper/2312.02116/) (2023)
- [Controlled Text Generation as Continuous Optimization with Multiple Constraints](https://mlsanity.com/paper/2108.01850/) (2021)
- [von Mises-Fisher Loss: An Exploration of Embedding Geometries for Supervised Learning](https://mlsanity.com/paper/2103.15718/) (2021)
- [Civil Rephrases Of Toxic Texts With Self-Supervised Transformers](https://mlsanity.com/paper/2102.05456/) (2021)
- [Machine Translation into Low-resource Language Varieties](https://mlsanity.com/paper/2106.06797/) (2021)
- [A Non-isotropic Probabilistic Take on Proxy-based Deep Metric Learning](https://mlsanity.com/paper/2207.03784/) (2022)
- [BERTTune: Fine-Tuning Neural Machine Translation with BERTScore](https://mlsanity.com/paper/2106.02208/) (2021)
- [Sparse Continuous Distributions and Fenchel-Young Losses](https://mlsanity.com/paper/2108.01988/) (2021)
- [To model human linguistic prediction, make LLMs less superhuman](https://mlsanity.com/paper/2510.05141/) (2025)
- [Graph Neural Networks for Low-Energy Event Classification & Reconstruction in IceCube](https://mlsanity.com/paper/2209.03042/) (2022)
- [Shedding a PAC-Bayesian Light on Adaptive Sliced-Wasserstein Distances](https://mlsanity.com/paper/2206.03230/) (2022)
- [Extending Multi-Sense Word Embedding to Phrases and Sentences for Unsupervised Semantic Applications](https://mlsanity.com/paper/2103.15330/) (2021)
- [On The Distribution of Penultimate Activations of Classification Networks](https://mlsanity.com/paper/2107.01900/) (2021)
- [No Shifted Augmentations (NSA): compact distributions for robust self-supervised Anomaly Detection](https://mlsanity.com/paper/2203.10344/) (2022)
- [Improving the Diversity of Unsupervised Paraphrasing with Embedding Outputs](https://mlsanity.com/paper/2110.13231/) (2021)
- [The Unreasonable Effectiveness of Random Target Embeddings for Continuous-Output Neural Machine Translation](https://mlsanity.com/paper/2310.20620/) (2023)
- [Unsupervised Transfer Learning in Multilingual Neural Machine Translation with Cross-Lingual Word Embeddings](https://mlsanity.com/paper/2103.06689/) (2021)
- [Keep your distance: learning dispersed embeddings on mathbbSm](https://mlsanity.com/paper/2502.08231/) (2025)
- [Unsupervised Discovery of Continuous Skills on a Sphere](https://mlsanity.com/paper/2305.14377/) (2023)
- [Explicit Word Density Estimation for Language Modelling](https://mlsanity.com/paper/2406.10256/) (2024)
- [Decoupled Vocabulary Learning Enables Zero-Shot Translation from Unseen Languages](https://mlsanity.com/paper/2408.02290/) (2024)
- [Deep Sparse Latent Feature Models for Knowledge Graph Completion](https://mlsanity.com/paper/2411.15694/) (2024)
- [Divergence-Based Similarity Function for Multi-View Contrastive Learning](https://mlsanity.com/paper/2507.06560/) (2025)
- [NuBench: An Open Benchmark for Deep Learning-Based Event Reconstruction in Neutrino Telescopes](https://mlsanity.com/paper/2511.13111/) (2025)
- [Representation Collapse in Machine Translation Through the Lens of Angular Dispersion](https://mlsanity.com/paper/2602.17287/) (2026)
- [Predicting build orientation for SLM dental parts: a comparison of rotation representations and direct vector regression](https://mlsanity.com/paper/2609.15710/) (2026)
- [Revealing Epistemic Uncertainty in MLLMs via Causal-Invariant Masking](https://mlsanity.com/paper/2610.02887/) (2026)

## Abstract (arXiv)

The Softmax function is used in the final layer of nearly all existing sequence-to-sequence models for language generation. However, it is usually the slowest layer to compute which limits the vocabulary size to a subset of most frequent types; and it has a large memory footprint. We propose a general technique for replacing the softmax layer with a continuous embedding layer. Our primary innovations are a novel probabilistic loss, and a training and inference procedure in which we generate a probability distribution over pre-trained word embeddings, instead of a multinomial distribution over the vocabulary obtained via softmax. We evaluate this new class of sequence-to-sequence models with continuous outputs on the task of neural machine translation. We show that our models obtain upto 2.5x speed-up in training time while performing on par with the state-of-the-art models in terms of translation quality. These models are capable of handling very large vocabularies without compromising on translation quality. They also produce more meaningful errors than in the softmax-based models, as these errors typically lie in a subspace of the vector space of the reference translations.