# Semantic Regularisation for Recurrent Image Annotation

arXiv: https://arxiv.org/abs/1611.05490 · HTML: https://mlsanity.com/paper/1611.05490/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2016.

## Built on by (most cited first, 14 of 14)

- [DualCoOp: Fast Adaptation to Multi-Label Recognition with Limited Annotations](https://mlsanity.com/paper/2206.09541/) (2022)
- [Transformer-based Dual Relation Graph for Multi-label Image Recognition](https://mlsanity.com/paper/2110.04722/) (2021)
- [Modeling Multi-Label Action Dependencies for Temporal Action Localization](https://mlsanity.com/paper/2103.03027/) (2021)
- [DualCoOp++: Fast and Effective Adaptation to Multi-Label Recognition with Limited Annotations](https://mlsanity.com/paper/2308.01890/) (2023)
- [Deep Learning for Multi-Label Learning: A Comprehensive Survey](https://mlsanity.com/paper/2401.16549/) (2024)
- [Multi-Label Image Classification with Contrastive Learning](https://mlsanity.com/paper/2107.11626/) (2021)
- [Graph Attention Transformer Network for Multi-Label Image Classification](https://mlsanity.com/paper/2203.04049/) (2022)
- [PLM: Partial Label Masking for Imbalanced Multi-label Classification](https://mlsanity.com/paper/2105.10782/) (2021)
- [Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning](https://mlsanity.com/paper/2403.01209/) (2024)
- [Improving Multi-label Recognition using Class Co-Occurrence Probabilities](https://mlsanity.com/paper/2404.16193/) (2024)
- [Rethinking Prompting Strategies for Multi-Label Recognition with Partial Annotations](https://mlsanity.com/paper/2409.08381/) (2024)
- [Visual Transformers with Primal Object Queries for Multi-Label Image Classification](https://mlsanity.com/paper/2112.05485/) (2021)
- [Reinforced Pedestrian Attribute Recognition with Group Optimization Reward](https://mlsanity.com/paper/2205.14042/) (2022)
- [Rethinking Multi-Label Image Classification With Deep Learning: Taxonomy, Challenge, and Outlook](https://mlsanity.com/paper/2607.00839/) (2026)

## Abstract (arXiv)

The "CNN-RNN" design pattern is increasingly widely applied in a variety of image annotation tasks including multi-label classification and captioning. Existing models use the weakly semantic CNN hidden layer or its transform as the image embedding that provides the interface between the CNN and RNN. This leaves the RNN overstretched with two jobs: predicting the visual concepts and modelling their correlations for generating structured annotation output. Importantly this makes the end-to-end training of the CNN and RNN slow and ineffective due to the difficulty of back propagating gradients through the RNN to train the CNN. We propose a simple modification to the design pattern that makes learning more effective and efficient. Specifically, we propose to use a semantically regularised embedding layer as the interface between the CNN and RNN. Regularising the interface can partially or completely decouple the learning problems, allowing each to be more effectively trained and jointly training much more efficient. Extensive experiments show that state-of-the art performance is achieved on multi-label classification as well as image captioning.