# Weakly supervised cross-domain alignment with optimal transport

arXiv: https://arxiv.org/abs/2008.06597 · HTML: https://mlsanity.com/paper/2008.06597/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2020.

## Built on by (most cited first, 5 of 5)

- [S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions](https://mlsanity.com/paper/2305.14095/) (2023)
- [VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment](https://mlsanity.com/paper/2210.04135/) (2022)
- [Can Brain Signals Reveal Inner Alignment with Human Languages?](https://mlsanity.com/paper/2208.06348/) (2022)
- [Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment](https://mlsanity.com/paper/2210.04722/) (2022)
- [LiveSeg: Unsupervised Multimodal Temporal Segmentation of Long Livestream Videos](https://mlsanity.com/paper/2210.05840/) (2022)

## Abstract (arXiv)

Cross-domain alignment between image objects and text sequences is key to many visual-language tasks, and it poses a fundamental challenge to both computer vision and natural language processing. This paper investigates a novel approach for the identification and optimization of fine-grained semantic similarities between image and text entities, under a weakly-supervised setup, improving performance over state-of-the-art solutions. Our method builds upon recent advances in optimal transport (OT) to resolve the cross-domain matching problem in a principled manner. Formulated as a drop-in regularizer, the proposed OT solution can be efficiently computed and used in combination with other existing approaches. We present empirical evidence to demonstrate the effectiveness of our approach, showing how it enables simpler model architectures to outperform or be comparable with more sophisticated designs on a range of vision-language tasks.