mlsanityData to 7 Oct 2026

SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models

arXiv 2311.07575 ↗First posted 13 Nov 2023cs.CV cs.AI cs.CLcited by 283

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We present SPHINX, a versatile multi-modal large language model (MLLM) with a joint mixing of model weights, tuning tasks, and visual embeddings. First, for stronger vision-language alignment, we unfreeze the large language model (LLM) during pre-training, and introduce a weight mix strategy between LLMs trained by real-world and synthetic data. By directly integrating the weights from two domains, the mixed LLM can efficiently incorporate diverse semantics with favorable robustness. Then, to enable multi-purpose capabilities, we mix a variety of tasks for joint visual instruction tuning, and design task-specific instructions to avoid inter-task conflict. In addition to the basic visual question answering, we include more challenging tasks such as region-level understanding, caption grounding, document layout detection, and human pose estimation, contributing to mutual enhancement over different scenarios. Additionally, we propose to extract comprehensive visual embeddings from various network architectures, pre-training paradigms, and information granularity, providing language models with more robust image representations. Based on our proposed joint mixing, SPHINX exhibits superior multi-modal understanding capabilities on a wide range of applications. On top of this, we further propose an efficient strategy aiming to better capture fine-grained appearances of high-resolution images. With a mixing of different scales and high-resolution sub-images, SPHINX attains exceptional visual parsing and reasoning performance on existing evaluation benchmarks. We hope our work may cast a light on the exploration of joint mixing in future MLLM research. Code is released at https://github.com/Alpha-VLLM/LLaMA2-Accessory.

Key facts. Cited by 283 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 74 of its 91 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

74 of 91 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Improving Compositional Text-to-image Generation with Large Vision-Language Models10 Oct
  2. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  3. 2023DreamLLM: Synergistic Multimodal Comprehension and Creation20 Sep
  4. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning14 Sep
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  11. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2023GPT-4 Technical Reportcited by 18,132
  9. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  10. 2022Training language models to follow instructions with human feedbackcited by 15,712
  11. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  12. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
38 more
  1. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  2. 2023Segment Anythingcited by 8,158
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  13. 2019Towards VQA Models That Can Readcited by 1,696
  14. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  15. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  16. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  17. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  18. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  19. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  20. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  21. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  22. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  23. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  24. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  25. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  26. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  27. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  28. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 730
  29. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  30. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  31. 2023Instruction Tuning with GPT-4cited by 676
  32. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  33. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  34. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  35. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  36. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  37. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  38. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 153; 2025: 93; 2026: 27.

283 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  5. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  6. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  7. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  8. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  9. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  10. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  11. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  12. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
38 more
  1. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  2. 2024What matters when building vision-language models?cited by 321
  3. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  4. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  5. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  6. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  7. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  8. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  9. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  10. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  11. 2024EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domaincited by 181
  12. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  13. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  14. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  15. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  16. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  17. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  18. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  19. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 148
  20. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  21. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  22. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  23. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  24. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  25. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  26. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  27. 2024Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Modelscited by 118
  28. 2023ChartBench: A Benchmark for Complex Visual Reasoning in Chartscited by 116
  29. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
  30. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  31. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 102
  32. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  33. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
  34. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
  35. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  36. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 82
  37. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  38. 2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79

50 most cited of 283 All 283 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023InfMLLM: A Unified Framework for Visual-Language Tasks
  2. 2025BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
  3. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
  4. 2025Improved Alignment of Modalities in Large Vision Language Models
  5. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy
  6. 2025MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding
  7. 2025Visual Representation Alignment for Multimodal Large Language Models
  8. 2024Cross-modal Information Flow in Multimodal Large Language Models