mlsanityData to 7 Oct 2026

Compositional Mixture Representations for Vision and Text

arXiv 2206.06404 ↗First posted 13 Jun 2022cs.CV cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Learning a common representation space between vision and language allows deep networks to relate objects in the image to the corresponding semantic meaning. We present a model that learns a shared Gaussian mixture representation imposing the compositionality of the text onto the visual domain without having explicit location supervision. By combining the spatial transformer with a representation learning approach we learn to split images into separately encoded patches to associate visual and textual representations in an interpretable manner. On variations of MNIST and CIFAR10, our model is able to perform weakly supervised object detection and demonstrates its ability to extrapolate to unseen combination of objects.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 13 Jun 2022.

Builds on

29 of 39 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2017Mask R-CNNcited by 7,259
  6. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  10. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  11. 2015Spatial Transformer Networkscited by 1,423
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
17 more
  1. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  2. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  3. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  4. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 388
  5. 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
  6. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  7. 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
  8. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  9. 2015Weakly Supervised Deep Detection Networkscited by 183
  10. 2017Dual-Path Convolutional Image-Text Embeddings with Instance Losscited by 145
  11. 2015Grounding of Textual Phrases in Images by Reconstructioncited by 112
  12. 2017The Consciousness Priorcited by 105
  13. 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
  14. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  15. 2017Towards Deeper Understanding of Variational Autoencoding Modelscited by 56
  16. 2020ALICE: Active Learning with Contrastive Natural Language Explanationscited by 36
  17. 2018Dissimilarity Coefficient based Weakly Supervised Object Detectioncited by 23

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 1; 2025: 1.

3 papers.

  1. 2022Disentangled Representation Learningcited by 111
  2. 2024LEARN: A Unified Framework for Multi-Task Domain Adapt Few-Shot Learningcited by 0
  3. 2025Modality-Collaborative Low-Rank Decomposers for Few-Shot Video Domain Adaptationcited by 0