mlsanityData to 7 Oct 2026

Large-Scale Bidirectional Training for Zero-Shot Image Captioning

arXiv 2211.06774 ↗First posted 13 Nov 2022cs.CV cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

When trained on large-scale datasets, image captioning models can understand the content of images from a general domain but often fail to generate accurate, detailed captions. To improve performance, pretraining-and-finetuning has been a key strategy for image captioning. However, we find that large-scale bidirectional training between image and text enables zero-shot image captioning. In this paper, we introduce Bidirectional Image Text Training in largER Scale, BITTERS, an efficient training and inference framework for zero-shot image captioning. We also propose a new evaluation benchmark which comprises of high quality datasets and an extensive set of metrics to properly evaluate zero-shot captioning accuracy and societal bias. We additionally provide an efficient finetuning approach for keyword extraction. We show that careful selection of large-scale training set and model architecture is the key to achieving zero-shot image captioning.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 54 of its 75 references that are in the map. First posted on arXiv 13 Nov 2022.

Builds on

54 of 75 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2017Decoupled Weight Decay Regularizationcited by 1,722
  10. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  13. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  15. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  16. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,199
  17. 2018Hierarchical Neural Story Generationcited by 1,182
  18. 2021Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Datacited by 961
  19. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  20. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  21. 2021GPT Understands, Toocited by 715
  22. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  23. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  24. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  25. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  26. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  27. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  28. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  29. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  30. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  31. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  32. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  33. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  34. 2015YFCC100M: The New Data in Multimedia Researchcited by 234
  35. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 196
  36. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  37. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  38. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023NICE: CVPR 2023 Challenge on Zero-shot Image Captioningcited by 9