# Learning Joint Representations of Videos and Sentences with Web Image Search

arXiv: https://arxiv.org/abs/1608.02367 · HTML: https://mlsanity.com/paper/1608.02367/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Aug 2016.

## Built on by (most cited first, 17 of 17)

- [UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection](https://mlsanity.com/paper/2203.12745/) (2022)
- [On Semantic Similarity in Video Retrieval](https://mlsanity.com/paper/2103.10095/) (2021)
- [Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding](https://mlsanity.com/paper/2203.02966/) (2022)
- [Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval](https://mlsanity.com/paper/2201.09168/) (2022)
- [EgoCVR: An Egocentric Benchmark for Fine-Grained Composed Video Retrieval](https://mlsanity.com/paper/2407.16658/) (2024)
- [Deep Learning for Video-Text Retrieval: a Review](https://mlsanity.com/paper/2302.12552/) (2023)
- [Multi-event Video-Text Retrieval](https://mlsanity.com/paper/2308.11551/) (2023)
- [Connecting Language and Vision for Natural Language-Based Vehicle Retrieval](https://mlsanity.com/paper/2105.14897/) (2021)
- [Dynamic Modeling of Hand-Object Interactions via Tactile Sensing](https://mlsanity.com/paper/2109.04378/) (2021)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)
- [Towards Retrieval Augmented Generation over Large Video Libraries](https://mlsanity.com/paper/2406.14938/) (2024)
- [Optimizing Latency for Online Video CaptioningUsing Audio-Visual Transformers](https://mlsanity.com/paper/2108.02147/) (2021)
- [Video-adverb retrieval with compositional adverb-action embeddings](https://mlsanity.com/paper/2309.15086/) (2023)
- [Sample Efficient Multimodal Semantic Augmentation for Incremental Summarization](https://mlsanity.com/paper/2303.04361/) (2023)
- [Query by Activity Video in the Wild](https://mlsanity.com/paper/2311.13895/) (2023)
- [Teleology-Driven Affective Computing: A Causal Framework for Sustained Well-Being](https://mlsanity.com/paper/2502.17172/) (2025)
- [USV: Towards Understanding the User-generated Short-form Videos](https://mlsanity.com/paper/2605.20838/) (2026)

## Abstract (arXiv)

Our objective is video retrieval based on natural language queries. In addition, we consider the analogous problem of retrieving sentences or generating descriptions given an input video. Recent work has addressed the problem by embedding visual and textual inputs into a common space where semantic similarities correlate to distances. We also adopt the embedding approach, and make the following contributions: First, we utilize web image search in sentence embedding process to disambiguate fine-grained visual concepts. Second, we propose embedding models for sentence, image, and video inputs whose parameters are learned simultaneously. Finally, we show how the proposed model can be applied to description generation. Overall, we observe a clear improvement over the state-of-the-art methods in the video and sentence retrieval tasks. In description generation, the performance level is comparable to the current state-of-the-art, although our embeddings were trained for the retrieval tasks.