# Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks

arXiv: https://arxiv.org/abs/1704.00260 · HTML: https://mlsanity.com/paper/1704.00260/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Apr 2017.

## Built on by (most cited first, 9 of 9)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Towards General Purpose Vision Systems](https://mlsanity.com/paper/2104.00743/) (2021)
- [UnionDet: Union-Level Detector Towards Real-Time Human-Object Interaction Detection](https://mlsanity.com/paper/2312.12664/) (2023)
- [Concept-skill Transferability-based Data Selection for Large Vision-Language Models](https://mlsanity.com/paper/2406.10995/) (2024)
- [Consistency Learning via Decoding Path Augmentation for Transformers in Human Object Interaction Detection](https://mlsanity.com/paper/2204.04836/) (2022)
- [Abstraction Alignment: Comparing Model-Learned and Human-Encoded Conceptual Relationships](https://mlsanity.com/paper/2407.12543/) (2024)
- [Domain-aware Self-supervised Pre-training for Label-Efficient Meme Analysis](https://mlsanity.com/paper/2209.14667/) (2022)
- [Context-Infused Visual Grounding for Art](https://mlsanity.com/paper/2410.12369/) (2024)
- [Semi-Supervised Domain Generalization for Object Detection via Language-Guided Feature Alignment](https://mlsanity.com/paper/2309.13525/) (2023)

## Abstract (arXiv)

An important goal of computer vision is to build systems that learn visual representations over time that can be applied to many tasks. In this paper, we investigate a vision-language embedding as a core representation and show that it leads to better cross-task transfer than standard multi-task learning. In particular, the task of visual recognition is aligned to the task of visual question answering by forcing each to use the same word-region embeddings. We show this leads to greater inductive transfer from recognition to VQA than standard multitask learning. Visual recognition also improves, especially for categories that have relatively few recognition training labels but appear often in the VQA setting. Thus, our paper takes a small step towards creating more general vision systems by showing the benefit of interpretable, flexible, and trainable core representations.