# Linking Image and Text with 2-Way Nets

arXiv: https://arxiv.org/abs/1608.07973 · HTML: https://mlsanity.com/paper/1608.07973/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2016.

## Built on by (most cited first, 14 of 14)

- [Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval](https://mlsanity.com/paper/2304.03391/) (2023)
- [SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text](https://mlsanity.com/paper/2204.11964/) (2022)
- [Telling the What while Pointing to the Where: Multimodal Queries for Image Retrieval](https://mlsanity.com/paper/2102.04980/) (2021)
- [You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval](https://mlsanity.com/paper/2403.07222/) (2024)
- [ASMR: Learning Attribute-Based Person Search with Adaptive Semantic Margin Regularizer](https://mlsanity.com/paper/2108.04533/) (2021)
- [UPAR: Unified Pedestrian Attribute Recognition and Person Retrieval](https://mlsanity.com/paper/2209.02522/) (2022)
- [Discrete-continuous Action Space Policy Gradient-based Attention for Image-Text Matching](https://mlsanity.com/paper/2104.10406/) (2021)
- [Towards Fast and Accurate Image-Text Retrieval with Self-Supervised Fine-Grained Alignment](https://mlsanity.com/paper/2308.14009/) (2023)
- [VLDeformer: Vision-Language Decomposed Transformer for Fast Cross-Modal Retrieval](https://mlsanity.com/paper/2110.11338/) (2021)
- [Boosting ship detection in SAR images with complementary pretraining techniques](https://mlsanity.com/paper/2103.08251/) (2021)
- [Attribute-Aware Implicit Modality Alignment for Text Attribute Person Search](https://mlsanity.com/paper/2406.03721/) (2024)
- [Learning to embed semantic similarity for joint image-text retrieval](https://mlsanity.com/paper/2210.03838/) (2022)
- [Active Mining Sample Pair Semantics for Image-text Matching](https://mlsanity.com/paper/2311.05425/) (2023)
- [Camouflage-aware Image-Text Retrieval via Expert Collaboration](https://mlsanity.com/paper/2604.01251/) (2026)

## Abstract (arXiv)

Linking two data sources is a basic building block in numerous computer vision problems. Canonical Correlation Analysis (CCA) achieves this by utilizing a linear optimizer in order to maximize the correlation between the two views. Recent work makes use of non-linear models, including deep learning techniques, that optimize the CCA loss in some feature space. In this paper, we introduce a novel, bi-directional neural network architecture for the task of matching vectors from two data sources. Our approach employs two tied neural network channels that project the two views into a common, maximally correlated space using the Euclidean loss. We show a direct link between the correlation-based loss and Euclidean loss, enabling the use of Euclidean loss for correlation maximization. To overcome common Euclidean regression optimization problems, we modify well-known techniques to our problem, including batch normalization and dropout. We show state of the art results on a number of computer vision matching tasks including MNIST image matching and sentence-image matching on the Flickr8k, Flickr30k and COCO datasets.