mlsanityData to 7 Oct 2026

Towards Multi-Modal Mastery: A 4.5B Parameter Truly Multi-Modal Small Language Model

arXiv 2411.05903 ↗First posted 8 Nov 2024cs.LG cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

We present a novel 4.5B parameter small language model that can handle multiple input and output modalities, including text, images, videos, and audio. Despite its small size, the model achieves near state-of-the-art performance on a variety of tasks, demonstrating the potential of multi-modal models to tackle complex real-world problems. Our approach leverages recent advancements in language modeling and multi-task learning to create a versatile and high-performing model that can even be deployed for edge inference. Experimental results show the model's strong performance across multiple benchmarks, paving the way for further progress in multi-modal artificial intelligence.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 27 of its 34 references that are in the map. First posted on arXiv 8 Nov 2024.

Builds on

27 of 34 references.

References · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
15 more
  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  2. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  5. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  6. 2024What matters when building vision-language models?cited by 321
  7. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  8. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  9. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  10. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 188
  11. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  12. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  13. 2022OCR-IDL: OCR Annotations for Industry Document Library Datasetcited by 35
  14. 2023OpenVoice: Versatile Instant Voice Cloningcited by 30
  15. 2024LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact Language Modelcited by 14

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22