# Multi-Modal Multi-Scale Deep Learning for Large-Scale Image Annotation

arXiv: https://arxiv.org/abs/1709.01220 · HTML: https://mlsanity.com/paper/1709.01220/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2017.

## Built on by (most cited first, 3 of 3)

- [SlowFast Rolling-Unrolling LSTMs for Action Anticipation in Egocentric Videos](https://mlsanity.com/paper/2109.00829/) (2021)
- [Motif Mining: Finding and Summarizing Remixed Image Content](https://mlsanity.com/paper/2203.08327/) (2022)
- [Dispersed Pixel Perturbation-based Imperceptible Backdoor Trigger for Image Classifier Models](https://mlsanity.com/paper/2208.09336/) (2022)

## Abstract (arXiv)

Image annotation aims to annotate a given image with a variable number of class labels corresponding to diverse visual concepts. In this paper, we address two main issues in large-scale image annotation: 1) how to learn a rich feature representation suitable for predicting a diverse set of visual concepts ranging from object, scene to abstract concept; 2) how to annotate an image with the optimal number of class labels. To address the first issue, we propose a novel multi-scale deep model for extracting rich and discriminative features capable of representing a wide range of visual concepts. Specifically, a novel two-branch deep neural network architecture is proposed which comprises a very deep main network branch and a companion feature fusion network branch designed for fusing the multi-scale features computed from the main branch. The deep model is also made multi-modal by taking noisy user-provided tags as model input to complement the image input. For tackling the second issue, we introduce a label quantity prediction auxiliary task to the main label prediction task to explicitly estimate the optimal label number for a given image. Extensive experiments are carried out on two large-scale image annotation benchmark datasets and the results show that our method significantly outperforms the state-of-the-art.