# Improving Text Generation with Student-Forcing Optimal Transport

arXiv: https://arxiv.org/abs/2010.05994 · HTML: https://mlsanity.com/paper/2010.05994/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Oct 2020.

## Built on by (most cited first, 8 of 8)

- [MiniLLM: On-Policy Distillation of Large Language Models](https://mlsanity.com/paper/2306.08543/) (2023)
- [Toward Interpretable Semantic Textual Similarity via Optimal Transport-based Contrastive Sentence Learning](https://mlsanity.com/paper/2202.13196/) (2022)
- [Re-evaluating Word Mover's Distance](https://mlsanity.com/paper/2105.14403/) (2021)
- [Bridging the Training-Inference Gap in LLMs by Leveraging Self-Generated Tokens](https://mlsanity.com/paper/2410.14655/) (2024)
- [Unbalanced Optimal Transport for Unbalanced Word Alignment](https://mlsanity.com/paper/2306.04116/) (2023)
- [OTSeq2Set: An Optimal Transport Enhanced Sequence-to-Set Model for Extreme Multi-label Text Classification](https://mlsanity.com/paper/2210.14523/) (2022)
- [Similarity Based Label Smoothing For Dialogue Generation](https://mlsanity.com/paper/2107.11481/) (2021)
- [Tracking Semantic Change in Slovene: A Novel Dataset and Optimal Transport-Based Distance](https://mlsanity.com/paper/2402.16596/) (2024)

## Abstract (arXiv)

Neural language models are often trained with maximum likelihood estimation (MLE), where the next word is generated conditioned on the ground-truth word tokens. During testing, however, the model is instead conditioned on previously generated tokens, resulting in what is termed exposure bias. To reduce this gap between training and testing, we propose using optimal transport (OT) to match the sequences generated in these two modes. An extension is further proposed to improve the OT learning, based on the structural and contextual information of the text sequences. The effectiveness of the proposed method is validated on machine translation, text summarization, and text generation tasks.