mlsanityData to 14 Oct 2026

S3: A Simple Strong Sample-effective Multimodal Dialog System

arXiv 2406.18305 ↗First posted 26 Jun 2024cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In this work, we present a conceptually simple yet powerful baseline for the multimodal dialog task, an S3 model, that achieves near state-of-the-art results on two compelling leaderboards: MMMU and AI Journey Contest 2023. The system is based on a pre-trained large language model, pre-trained modality encoders for image and audio, and a trainable modality projector. The proposed effective data mixture for training such an architecture demonstrates that a multimodal model based on a strong language model and trained on a small amount of multimodal data can perform efficiently in the task of multimodal dialog.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 30 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

27 of 30 references.

References · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  7. 2023Mistral 7Bcited by 3,164
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
15 more
  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  3. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  5. 2024Yi: Open Foundation Models by 01.AIcited by 719
  6. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  7. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  8. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  9. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  10. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
  11. 2016Visual Dialogcited by 377
  12. 2019Clotho: An Audio Captioning Datasetcited by 358
  13. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  14. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  15. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131

Built on by

Not cited yet.