# Object Captioning and Retrieval with Natural Language

arXiv: https://arxiv.org/abs/1803.06152 · HTML: https://mlsanity.com/paper/1803.06152/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Mar 2018.

## Built on by (most cited first, 6 of 6)

- [RSVG: Exploring Data and Models for Visual Grounding on Remote Sensing Data](https://mlsanity.com/paper/2210.12634/) (2022)
- [Coarse-to-Fine Reasoning for Visual Question Answering](https://mlsanity.com/paper/2110.02526/) (2021)
- [A Joint Network for Grasp Detection Conditioned on Natural Language Commands](https://mlsanity.com/paper/2104.00492/) (2021)
- [Graph-based Person Signature for Person Re-Identifications](https://mlsanity.com/paper/2104.06770/) (2021)
- [Light-weight Deformable Registration using Adversarial Learning with Distilling Knowledge](https://mlsanity.com/paper/2110.01293/) (2021)
- [Attention Gate in Traffic Forecasting](https://mlsanity.com/paper/2109.13021/) (2021)

## Abstract (arXiv)

We address the problem of jointly learning vision and language to understand the object in a fine-grained manner. The key idea of our approach is the use of object descriptions to provide the detailed understanding of an object. Based on this idea, we propose two new architectures to solve two related problems: object captioning and natural language-based object retrieval. The goal of the object captioning task is to simultaneously detect the object and generate its associated description, while in the object retrieval task, the goal is to localize an object given an input query. We demonstrate that both problems can be solved effectively using hybrid end-to-end CNN-LSTM networks. The experimental results on our new challenging dataset show that our methods outperform recent methods by a fair margin, while providing a detailed understanding of the object and having fast inference time. The source code will be made available.