SoupLM: Model Integration in Large Language and Multi-Modal Models
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Training large language models (LLMs) and multimodal LLMs necessitates significant computing resources, and existing publicly available LLMs are typically pre-trained on diverse, privately curated datasets spanning various tasks. For instance, LLaMA, Vicuna, and LLaVA are three LLM variants trained with LLaMA base models using very different training recipes, tasks, and data modalities. The training cost and complexity for such LLM variants grow rapidly. In this study, we propose to use a soup strategy to assemble these LLM variants into a single well-generalized multimodal LLM (SoupLM) in a cost-efficient manner. Assembling these LLM variants efficiently brings knowledge and specialities trained from different domains and data modalities into an integrated one (e.g., chatbot speciality from user-shared conversations for Vicuna, and visual capacity from vision-language data for LLaVA), therefore, to avoid computing costs of repetitive training on several different domains. We propose series of soup strategies to systematically benchmark performance gains across various configurations, and probe the soup behavior across base models in the interpolation space.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 56 references that are in the map. First posted on arXiv 11 Jul 2024.
Builds on
42 of 56 references.
Closest ancestors · 1
Same direction, newest first
Other references · 41
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2023GPT-4 Technical Reportcited by 18,132
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
29 more
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
- 2020Learning to summarize from human feedbackcited by 2,562
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
- 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
- 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
- 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
- 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
- 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
- 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
- 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
- 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
- 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
- 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
- 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
- 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
- 2024Evolutionary Optimization of Model Merging Recipescited by 204
- 2023Any-to-Any Generation via Composable Diffusioncited by 203
- 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques
- 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
- 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Survey
- 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
- 2024Enhancing Model Performance: Another Approach to Vision-Language Instruction Tuning
- 2024Model Composition for Multimodal Large Language Models
- 2026Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
- 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages