# Learning Shared Semantic Space with Correlation Alignment for Cross-modal Event Retrieval

arXiv: https://arxiv.org/abs/1901.04268 · HTML: https://mlsanity.com/paper/1901.04268/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2019.

## Built on by (most cited first, 3 of 3)

- [Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT](https://mlsanity.com/paper/2102.07594/) (2021)
- [Cross-Modal and Multimodal Data Analysis Based on Functional Mapping of Spectral Descriptors and Manifold Regularization](https://mlsanity.com/paper/2105.05631/) (2021)
- [QR-CLIP: Introducing Explicit Open-World Knowledge for Location and Time Reasoning](https://mlsanity.com/paper/2302.00952/) (2023)

## Abstract (arXiv)

In this paper, we propose to learn shared semantic space with correlation alignment (S3CA) for multimodal data representations, which aligns nonlinear correlations of multimodal data distributions in deep neural networks designed for heterogeneous data. In the context of cross-modal (event) retrieval, we design a neural network with convolutional layers and fully-connected layers to extract features for images, including images on Flickr-like social media. Simultaneously, we exploit a fully-connected neural network to extract semantic features for texts, including news articles from news media. In particular, nonlinear correlations of layer activations in the two neural networks are aligned with correlation alignment during the joint training of the networks. Furthermore, we project the multimodal data into a shared semantic space for cross-modal (event) retrieval, where the distances between heterogeneous data samples can be measured directly. In addition, we contribute a Wiki-Flickr Event dataset, where the multimodal data samples are not describing each other in pairs like the existing paired datasets, but all of them are describing semantic events. Extensive experiments conducted on both paired and unpaired datasets manifest the effectiveness of S3CA, outperforming the state-of-the-art methods.