mlsanityData to 7 Oct 2026

SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models

arXiv 2402.05935 ↗First posted 8 Feb 2024cs.CV cs.AI cs.CLcited by 130

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM) series developed upon SPHINX. To improve the architecture and training efficiency, we modify the SPHINX framework by removing redundant visual encoders, bypassing fully-padded sub-images with skip tokens, and simplifying multi-stage training into a one-stage all-in-one paradigm. To fully unleash the potential of MLLMs, we assemble a comprehensive multi-domain and multimodal dataset covering publicly available resources in language, vision, and vision-language tasks. We further enrich this collection with our curated OCR intensive and Set-of-Mark datasets, extending the diversity and generality. By training over different base LLMs including TinyLlama1.1B, InternLM2-7B, LLaMA2-13B, and Mixtral8x7B, we obtain a spectrum of MLLMs that vary in parameter size and multilingual capabilities. Comprehensive benchmarking reveals a strong correlation between the multi-modal performance with the data and parameter scales. Code and models are released at https://github.com/Alpha-VLLM/LLaMA2-Accessory

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 102 of its 126 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

102 of 126 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark22 Jan
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  3. 2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models5 Dec
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  5. 2023InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models20 Nov
  6. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  7. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V17 Oct
  8. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  9. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
2 more
  1. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 88

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2022A ConvNet for the 2020scited by 3,545
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  8. 2023Mistral 7Bcited by 3,065
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2015VQA: Visual Question Answeringcited by 2,489
  13. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  14. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  15. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  16. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  17. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  18. 2024Mixtral of Expertscited by 1,785
  19. 2019Towards VQA Models That Can Readcited by 1,696
  20. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  21. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  22. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  23. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  24. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  25. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  26. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  27. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  28. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  29. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  30. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  31. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  32. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  33. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  34. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  35. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  36. 2016A Diagram Is Worth A Dozen Imagescited by 793
  37. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  38. 2023Baichuan 2: Open Large-scale Language Modelscited by 664

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 80; 2025: 37; 2026: 13.

130 papers.

Sort:

By area:

  1. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  3. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  4. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  5. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  6. 2024What matters when building vision-language models?cited by 321
  7. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  8. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  9. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  10. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  11. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  12. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
38 more
  1. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  2. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  3. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  4. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  5. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  6. 2024Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiTcited by 130
  7. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 123
  8. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 120
  9. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  10. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
  11. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  12. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 102
  13. 2024Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Wantcited by 98
  14. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 98
  15. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  16. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  17. 2025Lumina-Image 2.0: A Unified and Efficient Image Generative Frameworkcited by 93
  18. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trendscited by 82
  19. 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
  20. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  21. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 72
  22. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  23. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  24. 2025Unlocking Multimodal Mathematical Reasoning via Process Reward Modelcited by 64
  25. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  26. 2024I2EBench: A Comprehensive Benchmark for Instruction-based Image Editingcited by 61
  27. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  28. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 55
  29. 2024Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Modelscited by 52
  30. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  31. 2024Dense Connector for MLLMscited by 48
  32. 2025MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoningcited by 45
  33. 2024Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Modelscited by 43
  34. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  35. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 43
  36. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 41
  37. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Modelscited by 37
  38. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37

50 most cited of 130 All 130 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  2. 2025NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints
  3. 2024MammothModa: Multi-Modal Large Language Model
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  5. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracy
  6. 2024EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model
  7. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  8. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models