# Deep Binaries: Encoding Semantic-Rich Cues for Efficient Textual-Visual Cross Retrieval

arXiv: https://arxiv.org/abs/1708.02531 · HTML: https://mlsanity.com/paper/1708.02531/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Aug 2017.

## Built on by (most cited first, 3 of 3)

- [LInK: Learning Joint Representations of Design and Performance Spaces through Contrastive Learning for Mechanism Synthesis](https://mlsanity.com/paper/2405.20592/) (2024)
- [A Discriminative Vectorial Framework for Multi-modal Feature Representation](https://mlsanity.com/paper/2103.05597/) (2021)
- [Highlight Timestamp Detection Model for Comedy Videos via Multimodal Sentiment Analysis](https://mlsanity.com/paper/2106.00451/) (2021)

## Abstract (arXiv)

Cross-modal hashing is usually regarded as an effective technique for large-scale textual-visual cross retrieval, where data from different modalities are mapped into a shared Hamming space for matching. Most of the traditional textual-visual binary encoding methods only consider holistic image representations and fail to model descriptive sentences. This renders existing methods inappropriate to handle the rich semantics of informative cross-modal data for quality textual-visual search tasks. To address the problem of hashing cross-modal data with semantic-rich cues, in this paper, a novel integrated deep architecture is developed to effectively encode the detailed semantics of informative images and long descriptive sentences, named as Textual-Visual Deep Binaries (TVDB). In particular, region-based convolutional networks with long short-term memory units are introduced to fully explore image regional details while semantic cues of sentences are modeled by a text convolutional network. Additionally, we propose a stochastic batch-wise training routine, where high-quality binary codes and deep encoding functions are efficiently optimized in an alternating manner. Experiments are conducted on three multimedia datasets, i.e. Microsoft COCO, IAPR TC-12, and INRIA Web Queries, where the proposed TVDB model significantly outperforms state-of-the-art binary coding methods in the task of cross-modal retrieval.