Representation Alignment from a Few Paired Data via Distilled Optimal Transport
Not on the map. Why?
Abstract (arXiv)
Aligning two independently pretrained unimodal encoders typically demands millions of paired examples. In this paper, we study the regime where only N ~ 100--500 pairs are available, while unpaired data on either side is abundant. Our method expresses every representation by its similarities to the N paired anchors, which places both modalities in a common coordinate system; matches the two point clouds with a single entropic optimal transport problem solved over the entire pool, whose marginal constraints counteract the hubness that defeats nearest-neighbor matching; and distills the resulting plan into two lightweight encoders, so that the correspondence becomes inductive. We additionally put this marginal constraint in relation with the one-step hub corrections of the bilingual lexicon induction literature. Pairing is required only for the anchors: the unpaired collections may be drawn independently from the two marginals, so that no item outside the anchor set needs to have both of its modalities observed. We evaluate on image-text and molecule-text alignment, where our method outperforms existing approaches at the same anchor budget on Flickr30k, MS-COCO and ChEBI-20.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 30 of its 35 references that are in the map. First posted on arXiv 8 Oct 2026.
Builds on
30 of 35 references.
References · 30
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
- 2018Computational Optimal Transportcited by 780
- 2021Designing an Encoder for StyleGAN Image Manipulationcited by 544
- 2017Word Translation Without Parallel Datacited by 487
- 2024The Platonic Representation Hypothesiscited by 382
- 2021Large-Scale Chemical Language Representations Capture Molecular Structure and Propertiescited by 206
- 2022Relative representations enable zero-shot latent space communicationcited by 191
- 2018Optimal Transport for structured data with application on graphscited by 172
18 more
- 2018Gromov-Wasserstein Alignment of Word Embedding Spacescited by 122
- 2018Unsupervised Alignment of Embeddings with Wasserstein Procrustescited by 97
- 2017Offline bilingual word vectors, orthogonal transformations and the inverted softmaxcited by 96
- 2021Cross Modal Retrieval with Querybank Normalisationcited by 72
- 2014Improving zero-shot learning by mitigating the hubness problemcited by 54
- 2022ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Trainingcited by 47
- 2026Revisiting the Platonic Representation Hypothesis: An Aristotelian Viewcited by 44
- 2021Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillationcited by 36
- 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 30
- 2025With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide Youcited by 16
- 2024CSA: Data-efficient Mapping of Unimodal Features to Multimodal Featurescited by 12
- 2023Bootstrapping Parallel Anchors for Relative Representationscited by 8
- 2023Weakly Supervised Vision-and-Language Pre-training with Relative Representationscited by 5
- 2025Learning Shared Representations from Unpaired Datacited by 4
- 2026SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transportcited by 3
- 2025Hubness Reduction with Dual Bank Sinkhorn Normalization for Cross-Modal Retrievalcited by 2
- 2025Model alignment using inter-modal bridgescited by 1
- 2026Learning Relative Representations for Fine-Grained Multimodal Alignment with Limited Datacited by 1
Built on by
Not cited yet.