mlsanityData to 14 Oct 2026

The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer

arXiv 2504.10462 ↗First posted 14 Apr 2025cs.CVcited by 30

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

This paper introduces SAIL, a single transformer unified multimodal large language model (MLLM) that integrates raw pixel encoding and language decoding within a singular architecture. Unlike existing modular MLLMs, which rely on a pre-trained vision transformer (ViT), SAIL eliminates the need for a separate vision encoder, presenting a more minimalist architecture design. Instead of introducing novel architectural components, SAIL adapts mix-attention mechanisms and multimodal positional encodings to better align with the distinct characteristics of visual and textual modalities. We systematically compare SAIL's properties-including scalability, cross-modal information flow patterns, and visual representation capabilities-with those of modular MLLMs. By scaling both training data and model size, SAIL achieves performance comparable to modular MLLMs. Notably, the removal of pretrained ViT components enhances SAIL's scalability and results in significantly different cross-modal information flow patterns. Moreover, SAIL demonstrates strong visual representation capabilities, achieving results on par with ViT-22B in vision tasks such as semantic segmentation. Code and models are available at https://github.com/bytedance/SAIL.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 72 of its 87 references that are in the map. First posted on arXiv 14 Apr 2025.

Builds on

72 of 87 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data24 Oct
  2. 2024Reconstructive Visual Instruction Tuning12 Oct
  3. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training10 Oct
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  5. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  7. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs11 Jan
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  10. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  12. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
2 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2023Mistral 7Bcited by 3,164
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
38 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  2. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  5. 2019Towards VQA Models That Can Readcited by 1,718
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  10. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  12. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  13. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  14. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  15. 2016A Diagram Is Worth A Dozen Imagescited by 804
  16. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  17. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  18. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  19. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  20. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  21. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  22. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  23. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  24. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  25. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  26. 2022Vision Transformer Adapter for Dense Predictionscited by 565
  27. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  28. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  29. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  30. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  31. 2024Scalable Pre-training of Large Autoregressive Image Modelscited by 115
  32. 2024Unveiling Encoder-Free Vision-Language Modelscited by 94
  33. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  34. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  35. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 87
  36. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 86
  37. 2023An Inverse Scaling Law for CLIP Trainingcited by 77
  38. 2023Hard Patches Mining for Masked Image Modelingcited by 57

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 13.

30 papers.

Sort:

By area:

  1. 2025VGR: Visual Grounded Reasoningcited by 73
  2. 2025Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodologycited by 69
  3. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  4. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 32
  5. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scalecited by 18
  6. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  7. 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 18
  8. 2025Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Modelscited by 14
  9. 2025Scheduling Your LLM Reinforcement Learning with Reasoning Treescited by 12
  10. 2025Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuningcited by 12
  11. 2025SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Modelcited by 10
  12. 2025NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraintscited by 6
18 more
  1. 2025SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinementcited by 4
  2. 2025PPE: Positional Preservation Embedding for Token Compression in Multimodal Large Language Modelscited by 4
  3. 2025AdaLRS: Loss-Guided Adaptive Learning Rate Search for Efficient Foundation Model Pretrainingcited by 3
  4. 2025RL makes MLLMs see better than SFTcited by 3
  5. 2025Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Trainingcited by 2
  6. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Modelscited by 2
  7. 2026Let ViT Speak: Generative Language-Image Pre-trainingcited by 2
  8. 2026Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoningcited by 2
  9. 2025SVAC: Scaling Is All You Need For Referring Video Object Segmentationcited by 1
  10. 2026PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Modelscited by 1
  11. 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generationcited by 1
  12. 2026TADS: Task-Aware Data Selection for Multi-Task Multimodal Pre-Trainingcited by 0
  13. 2026Early Intervention for VFM-based Multimodal Medical Image Classificationcited by 0
  14. 2026SigLIP-HD by Fine-to-Coarse Supervisioncited by 0
  15. 2026CAPEval: A Decoupled Caption Evaluation across Understanding and Generationcited by 0
  16. 2026MuRA: Multi-Rank Adaptation for Efficient and Effective Test-Time Vision-Language Generalizationcited by 0
  17. 2026How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretrainingcited by 0
  18. 2026Architecture-Dependent Fusion Pathways in MLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding
  2. 2026VersaViT: Enhancing MLLM Vision Backbones via Task-Guided Optimization
  3. 2025Rethinking Visual Information Processing in Multimodal LLMs
  4. 2026ParVL: Parallel Scaling and Expandable Compute Allocation for Multimodal LLMs
  5. 2025InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
  6. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE
  7. 2024Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model
  8. 2026A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models