mlsanityData to 14 Oct 2026

Multimodal Knowledge Alignment with Reinforcement Learning

arXiv 2205.12630 ↗First posted 25 May 2022cs.CL cs.CVcited by 30

Not on the map. Why?

Abstract (arXiv)

Large language models readily adapt to novel settings, even without task-specific training data. Can their zero-shot capacity be extended to multimodal inputs? In this work, we propose ESPER which extends language-only zero-shot models to unseen multimodal tasks, like image and audio captioning. Our key novelty is to use reinforcement learning to align multimodal inputs to language model generations without direct supervision: for example, in the image case our reward optimization relies only on cosine similarity derived from CLIP, and thus requires no additional explicitly paired (image, caption) data. Because the parameters of the language model are left unchanged, the model maintains its capacity for zero-shot generalization. Experiments demonstrate that ESPER outperforms baselines and prior work on a variety of zero-shot tasks; these include a new benchmark we collect+release, ESP dataset, which tasks models with generating several diversely-styled captions for each image.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 63 of its 85 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

63 of 85 references.

References · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2017Proximal Policy Optimization Algorithmscited by 15,215
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,916
38 more
  1. 2020Learning to summarize from human feedbackcited by 2,588
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  3. 2015VQA: Visual Question Answeringcited by 2,506
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  12. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  13. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  14. 2021GPT Understands, Toocited by 716
  15. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  16. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  17. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  18. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  19. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  20. 2016Self-critical Sequence Training for Image Captioningcited by 547
  21. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  22. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  23. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  24. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 417
  25. 2016Visual Dialogcited by 377
  26. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  27. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 327
  28. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  29. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  30. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 204
  31. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  32. 2016Visual Storytellingcited by 176
  33. 2019Interactive Fiction Games: A Colossal Adventurecited by 168
  34. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  35. 2020Visual News: Benchmark and Challenges in News Image Captioningcited by 121
  36. 2019Counterfactual Story Reasoning and Generationcited by 88
  37. 2018Unsupervised Image Captioningcited by 79
  38. 2019Good News, Everyone! Context driven entity-aware captioning for news imagescited by 66

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 13; 2024: 11; 2025: 2.

30 papers.

Sort:

By area:

  1. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  2. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187
  3. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  4. 2022SuS-X: Training-Free Name-Only Transfer of Vision-Language Modelscited by 119
  5. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  6. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  7. 2023Interactive Natural Language Processingcited by 31
  8. 2024Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Datacited by 27
  9. 2022I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervisioncited by 26
  10. 2023DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4cited by 16
  11. 2024Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Trainingcited by 16
  12. 2023CHAMPAGNE: Learning Real-world Conversation from Large-Scale Web Videoscited by 14
18 more
  1. 2023Cross-Domain Image Captioning with Discriminative Finetuningcited by 14
  2. 2023Improving Cross-modal Alignment with Synthetic Pairs for Text-only Image Captioningcited by 14
  3. 2024Multimodal LLM Enhanced Cross-lingual Cross-modal Retrievalcited by 13
  4. 2024Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Trainingcited by 12
  5. 2024Bridging Vision and Language Spaces with Assignment Predictioncited by 11
  6. 2024What Makes for Good Image Captions?cited by 11
  7. 2023CgT-GAN: CLIP-guided Text GAN for Image Captioningcited by 10
  8. 2025MIRAGE: Multimodal Immersive Reasoning and Guided Exploration for Red-Team Jailbreak Attackscited by 10
  9. 2023Image Captioning with Multi-Context Synthetic Datacited by 7
  10. 2025From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systemscited by 6
  11. 2024ArcSin: Adaptive ranged cosine Similarity injected noise for Language-Driven Visual Taskscited by 4
  12. 2024Distinctive Image Captioning: Leveraging Ground Truth Captions in CLIP Guided Reinforcement Learningcited by 2
  13. 2024Hierarchical Meta-Reinforcement Learning via Automated Macro-Action Discoverycited by 2
  14. 2023VLIS: Unimodal Language Models Guide Multimodal Language Generationcited by 1
  15. 2023Boosting Audio-visual Zero-shot Learning with Large Language Modelscited by 1
  16. 2023Linear Alignment of Vision-language Models for Image Captioningcited by 0
  17. 2024Experimental Design for Active Transductive Inference in Large Language Modelscited by 0
  18. 2024Fluent and Accurate Image Captioning with a Self-Trained Reward Modelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback
  2. 2026LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generation
  3. 2023On the Performance of Multimodal Language Models
  4. 2023Apollo: Zero-shot MultiModal Reasoning with Multiple Experts
  5. 2025Aligning Multimodal LLM with Human Preference: A Survey
  6. 2025UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning
  7. 2025PairUni: Pairwise Training for Unified Multimodal Language Models
  8. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioning