# Compositional Mixture Representations for Vision and Text

arXiv: https://arxiv.org/abs/2206.06404 · HTML: https://mlsanity.com/paper/2206.06404/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 13 Jun 2022.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Multi-Object Representation Learning with Iterative Variational Inference](https://mlsanity.com/paper/1903.00450/) (2019)
- [How Much Can CLIP Benefit Vision-and-Language Tasks?](https://mlsanity.com/paper/2107.06383/) (2021)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Weakly Supervised Deep Detection Networks](https://mlsanity.com/paper/1511.02853/) (2015)
- [Dual-Path Convolutional Image-Text Embeddings with Instance Loss](https://mlsanity.com/paper/1711.05535/) (2017)
- [Grounding of Textual Phrases in Images by Reconstruction](https://mlsanity.com/paper/1511.03745/) (2015)
- [The Consciousness Prior](https://mlsanity.com/paper/1709.08568/) (2017)
- [Deep Fragment Embeddings for Bidirectional Image Sentence Mapping](https://mlsanity.com/paper/1406.5679/) (2014)
- [WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training](https://mlsanity.com/paper/2103.06561/) (2021)
- [Towards Deeper Understanding of Variational Autoencoding Models](https://mlsanity.com/paper/1702.08658/) (2017)
- [ALICE: Active Learning with Contrastive Natural Language Explanations](https://mlsanity.com/paper/2009.10259/) (2020)
- [Dissimilarity Coefficient based Weakly Supervised Object Detection](https://mlsanity.com/paper/1811.10016/) (2018)

## Built on by (most cited first, 3 of 3)

- [Disentangled Representation Learning](https://mlsanity.com/paper/2211.11695/) (2022)
- [LEARN: A Unified Framework for Multi-Task Domain Adapt Few-Shot Learning](https://mlsanity.com/paper/2412.16275/) (2024)
- [Modality-Collaborative Low-Rank Decomposers for Few-Shot Video Domain Adaptation](https://mlsanity.com/paper/2511.18711/) (2025)

## Abstract (arXiv)

Learning a common representation space between vision and language allows deep networks to relate objects in the image to the corresponding semantic meaning. We present a model that learns a shared Gaussian mixture representation imposing the compositionality of the text onto the visual domain without having explicit location supervision. By combining the spatial transformer with a representation learning approach we learn to split images into separately encoded patches to associate visual and textual representations in an interpretable manner. On variations of MNIST and CIFAR10, our model is able to perform weakly supervised object detection and demonstrates its ability to extrapolate to unseen combination of objects.