# Improving Sequence-to-Sequence Learning via Optimal Transport

arXiv: https://arxiv.org/abs/1901.06283 · HTML: https://mlsanity.com/paper/1901.06283/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jan 2019.

## Built on by (most cited first, 30 of 33)

- [PLOT: Prompt Learning with Optimal Transport for Vision-Language Models](https://mlsanity.com/paper/2210.01253/) (2022)
- [A Survey on Optimal Transport for Machine Learning: Theory and Applications](https://mlsanity.com/paper/2106.01963/) (2021)
- [Pre-training for Speech Translation: CTC Meets Optimal Transport](https://mlsanity.com/paper/2301.11716/) (2023)
- [Improving Neural Cross-Lingual Summarization via Employing Optimal Transport Distance for Knowledge Distillation](https://mlsanity.com/paper/2112.03473/) (2021)
- [VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment](https://mlsanity.com/paper/2210.04135/) (2022)
- [Using Optimal Transport as Alignment Objective for fine-tuning Multilingual Contextualized Embeddings](https://mlsanity.com/paper/2110.02887/) (2021)
- [Fine-Tuning Graph Neural Networks via Graph Topology induced Optimal Transport](https://mlsanity.com/paper/2203.10453/) (2022)
- [vONTSS: vMF based semi-supervised neural topic modeling with optimal transport](https://mlsanity.com/paper/2307.01226/) (2023)
- [Improving Cross-lingual Information Retrieval on Low-Resource Languages via Optimal Transport Distillation](https://mlsanity.com/paper/2301.12566/) (2023)
- [Pushing the Limits of Zero-shot End-to-End Speech Translation](https://mlsanity.com/paper/2402.10422/) (2024)
- [Re-evaluating Word Mover's Distance](https://mlsanity.com/paper/2105.14403/) (2021)
- [Innovations in Neural Data-to-text Generation: A Survey](https://mlsanity.com/paper/2207.12571/) (2022)
- [Functional optimal transport: map estimation and domain adaptation for functional data](https://mlsanity.com/paper/2102.03895/) (2021)
- [Deep Cut-informed Graph Embedding and Clustering](https://mlsanity.com/paper/2503.06635/) (2025)
- [Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment](https://mlsanity.com/paper/2210.04722/) (2022)
- [LiveSeg: Unsupervised Multimodal Temporal Segmentation of Long Livestream Videos](https://mlsanity.com/paper/2210.05840/) (2022)
- [Unbalanced Optimal Transport for Unbalanced Word Alignment](https://mlsanity.com/paper/2306.04116/) (2023)
- [Enhancing Unsupervised Graph Few-shot Learning via Set Functions and Optimal Transport](https://mlsanity.com/paper/2501.05635/) (2025)
- [Cross-Modality Earth Mover's Distance for Visible Thermal Person Re-Identification](https://mlsanity.com/paper/2203.01675/) (2022)
- [Expand BERT Representation with Visual Information via Grounded Language Learning with Multimodal Partial Alignment](https://mlsanity.com/paper/2312.01592/) (2023)
- [SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2412.11026/) (2024)
- [Data Augmentation for Text Generation Without Any Augmented Data](https://mlsanity.com/paper/2105.13650/) (2021)
- [Nested-Wasserstein Self-Imitation Learning for Sequence Generation](https://mlsanity.com/paper/2001.06944/) (2020)
- [OTSeq2Set: An Optimal Transport Enhanced Sequence-to-Set Model for Extreme Multi-label Text Classification](https://mlsanity.com/paper/2210.14523/) (2022)
- [Improving word mover's distance by leveraging self-attention matrix](https://mlsanity.com/paper/2211.06229/) (2022)
- [KDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillation](https://mlsanity.com/paper/2307.01878/) (2023)
- [Adversarial Machine Learning in Text Analysis and Generation](https://mlsanity.com/paper/2101.08675/) (2021)
- [Semi-constraint Optimal Transport for Entity Alignment with Dangling Cases](https://mlsanity.com/paper/2203.05744/) (2022)
- [Unsupervised Opinion Summarisation in the Wasserstein Space](https://mlsanity.com/paper/2211.14923/) (2022)
- [Exploring Wasserstein Distance across Concept Embeddings for Ontology Matching](https://mlsanity.com/paper/2207.11324/) (2022)

## Abstract (arXiv)

Sequence-to-sequence models are commonly trained via maximum likelihood estimation (MLE). However, standard MLE training considers a word-level objective, predicting the next word given the previous ground-truth partial sentence. This procedure focuses on modeling local syntactic patterns, and may fail to capture long-range semantic structure. We present a novel solution to alleviate these issues. Our approach imposes global sequence-level guidance via new supervision based on optimal transport, enabling the overall characterization and preservation of semantic features. We further show that this method can be understood as a Wasserstein gradient flow trying to match our model to the ground truth sequence distribution. Extensive experiments are conducted to validate the utility of the proposed approach, showing consistent improvements over a wide variety of NLP tasks, including machine translation, abstractive text summarization, and image captioning.