mlsanityData to 7 Oct 2026

Metropolis-Hastings Captioning Game: Knowledge Fusion of Vision Language Models via Decentralized Bayesian Inference

arXiv 2504.09620 ↗First posted 13 Apr 2025cs.CL cs.AI cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We propose the Metropolis-Hastings Captioning Game (MHCG), a method to fuse knowledge of multiple vision-language models (VLMs) by learning from each other. Although existing methods that combine multiple models suffer from inference costs and architectural constraints, MHCG avoids these problems by performing decentralized Bayesian inference through a process resembling a language game. The knowledge fusion process establishes communication between two VLM agents alternately captioning images and learning from each other. We conduct two image-captioning experiments with two VLMs, each pre-trained on a different dataset. The first experiment demonstrates that MHCG achieves consistent improvement in reference-free evaluation metrics. The second experiment investigates how MHCG contributes to sharing VLMs' category-level vocabulary by observing the occurrence of the vocabulary in the generated captions.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 55 references that are in the map. First posted on arXiv 13 Apr 2025.

Builds on

33 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  9. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  10. 2020Federated Learning with Matched Averagingcited by 564
  11. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  12. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
20 more
  1. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  2. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  3. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  4. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  5. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  6. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  7. 2016Multi-Agent Cooperation and the Emergence of (Natural) Languagecited by 181
  8. 2020Emergent Multi-Agent Communication in the Deep Learning Eracited by 168
  9. 2017Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbolscited by 123
  10. 2020Compositionality and Generalization in Emergent Languagescited by 103
  11. 2017Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialogcited by 86
  12. 2023Positive-Augmented Contrastive Learning for Image and Video Captioning Evaluationcited by 78
  13. 2019Anti-efficient encoding in emergent communicationcited by 59
  14. 2023Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltycited by 57
  15. 2024Pack of LLMs: Model Fusion at Test-Time via Perplexity Optimizationcited by 56
  16. 2024ALOHa: A New Measure for Hallucination in Captioning Modelscited by 44
  17. 2024A Review of the Applications of Deep Learning-Based Emergent Communicationcited by 17
  18. 2023Cross-Domain Image Captioning with Discriminative Finetuningcited by 14
  19. 2023Referential communication in heterogeneous communities of pre-trained visual deep networkscited by 10
  20. 2023Recursive Metropolis-Hastings Naming Game: Symbol Emergence in a Multi-agent System based on Probabilistic Generative Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Emergent Communication between Heterogeneous Visual Agents through Decentralized Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  2. 2024LaMI: Augmenting Large Language Models via Late Multi-Image Fusion
  3. 2025Emergent Natural Language with Communication Games for Improving Image Captioning Capabilities without Additional Data
  4. 2025Top-Down Semantic Refinement for Image Captioning
  5. 2022Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles
  6. 2026BalCapRL: A Balanced Framework for RL-Based MLLM Image Captioning
  7. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning
  8. 2026Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning