mlsanityData to 13 Oct 2026

LAViTeR: Learning Aligned Visual and Textual Representations Assisted by Image and Caption Generation

arXiv 2109.04993 ↗First posted 4 Sep 2021cs.CV cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Pre-training visual and textual representations from large-scale image-text pairs is becoming a standard approach for many downstream vision-language tasks. The transformer-based models learn inter and intra-modal attention through a list of self-supervised learning tasks. This paper proposes LAViTeR, a novel architecture for visual and textual representation learning. The main module, Visual Textual Alignment (VTA) will be assisted by two auxiliary tasks, GAN-based image synthesis and Image Captioning. We also propose a new evaluation metric measuring the similarity between the learnt visual and textual embedding. The experimental results on two public datasets, CUB and MS-COCO, demonstrate superior visual and textual representation alignment in the joint feature embedding space

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 38 of its 59 references that are in the map. First posted on arXiv 4 Sep 2021.

Builds on

38 of 59 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,106
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,625
  8. 2020End-to-End Object Detection with Transformerscited by 7,396
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,457
  10. 2016Improved Techniques for Training GANscited by 2,946
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,810
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,220
26 more
  1. 2014Conditional Generative Adversarial Netscited by 2,157
  2. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,035
  3. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,551
  4. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  6. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,294
  7. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  8. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 878
  10. 2016Generative Adversarial Text to Image Synthesiscited by 735
  11. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  12. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  13. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  14. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  15. 2019Attention on Attention for Image Captioningcited by 289
  16. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  17. 2018Exploring Visual Relationship for Image Captioningcited by 215
  18. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  19. 2019Image Captioning: Transforming Objects into Wordscited by 171
  20. 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
  21. 2014From Captions to Visual Concepts and Backcited by 109
  22. 2019Fusion of Detected Objects in Text for Visual Question Answeringcited by 74
  23. 2016Instance-aware Image and Sentence Matching with Selective Multimodal LSTMcited by 43
  24. 2020Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessmentcited by 37
  25. 2020A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reportscited by 35
  26. 2020Image Captioning through Image Transformercited by 29

Built on by

Not cited yet.