mlsanityData to 14 Oct 2026

BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices

arXiv 2411.10640 ↗First posted 16 Nov 2024cs.CV cs.CLcited by 25

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The emergence and growing popularity of multimodal large language models (MLLMs) have significant potential to enhance various aspects of daily life, from improving communication to facilitating learning and problem-solving. Mobile phones, as essential daily companions, represent the most effective and accessible deployment platform for MLLMs, enabling seamless integration into everyday tasks. However, deploying MLLMs on mobile phones presents challenges due to limitations in memory size and computational capability, making it difficult to achieve smooth and real-time processing without extensive optimization. In this paper, we present BlueLM-V-3B, an algorithm and system co-design approach specifically tailored for the efficient deployment of MLLMs on mobile platforms. To be specific, we redesign the dynamic resolution scheme adopted by mainstream MLLMs and implement system optimization for hardware-aware deployment to optimize model inference on mobile phones. BlueLM-V-3B boasts the following key highlights: (1) Small Size: BlueLM-V-3B features a language model with 2.7B parameters and a vision encoder with 400M parameters. (2) Fast Speed: BlueLM-V-3B achieves a generation speed of 24.4 token/s on the MediaTek Dimensity 9300 processor with 4-bit LLM weight quantization. (3) Strong Performance: BlueLM-V-3B has attained the highest average score of 66.1 on the OpenCompass benchmark among models with ≤ 4B parameters and surpassed a series of models with much larger parameter sizes (e.g., MiniCPM-V-2.6, InternVL2-8B).

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 68 of its 147 references that are in the map. First posted on arXiv 16 Nov 2024.

Builds on

68 of 147 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training10 Oct
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  3. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  5. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  7. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  9. 2023UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model8 Oct
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  11. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun

Other references · 57

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  12. 2023PaLM 2 Technical Reportcited by 1,162
38 more
  1. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  2. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  3. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  4. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  6. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  7. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  8. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  9. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  10. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 475
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  12. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  13. 2023PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answeringcited by 338
  14. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
  15. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  16. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  17. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 236
  18. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  19. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 205
  20. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 174
  21. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 169
  22. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  23. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  24. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  25. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 150
  26. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 144
  27. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Datasetcited by 143
  28. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139
  29. 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expressioncited by 137
  30. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  31. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  32. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  33. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 100
  34. 2022MapQA: A Dataset for Question Answering on Choropleth Mapscited by 85
  35. 2024MultiMath: Bridging Visual and Mathematical Reasoning for Large Language Modelscited by 70
  36. 2023SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphscited by 68
  37. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 66
  38. 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 55

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 19; 2026: 5.

25 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  3. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
  4. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 74
  5. 2025Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Modelscited by 32
  6. 2025Ocean-OCR: Towards General OCR Application via a Vision-Language Modelcited by 25
  7. 2025Valley2: Exploring Multimodal Models with Scalable Vision-Language Designcited by 21
  8. 2025Scaling LLM Test-Time Compute with Mobile NPU on Smartphonescited by 11
  9. 2026PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarioscited by 10
  10. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7
  11. 2025EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devicescited by 5
  12. 2025BlueLM-2.5-3B Technical Reportcited by 5
13 more
  1. 2025Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Executioncited by 5
  2. 2025AutoJudger: An Agent-Driven Framework for Efficient Benchmarking of MLLMscited by 4
  3. 2025AndesVL Technical Report: An Efficient Mobile-side Multimodal Large Language Modelcited by 4
  4. 2025GenieBlue: Integrating both Linguistic and Multimodal Capabilities for Large Language Models on Mobile Devicescited by 3
  5. 2025SmartBench: Is Your LLM Truly a Good Chinese Smartphone Assistant?cited by 2
  6. 2025Evaluating Small Vision-Language Models on Distance-Dependent Traffic Perceptioncited by 2
  7. 2025EdgeInfinite-Instruct: Bridging SFT-Based Optimization and NPU-Level Efficiency for Edge Devicescited by 1
  8. 2025MagicVL-2B: Empowering Vision-Language Models on Mobile Devices with Lightweight Visual Encoders via Curriculum Learningcited by 0
  9. 2025An Automated Multi-modal Evaluation Framework for Mobile Intelligent Assistants Based on Large Language Models and Multi-Agent Collaborationcited by 0
  10. 2025Beyond One-Way Pruning: Bidirectional Pruning-Regrowth for Extreme Accuracy-Sparsity Tradeoffcited by 0
  11. 2026VPRune: Efficient Training-free Pre-LLM Visual Token Pruningcited by 0
  12. 2026Empowering Hybrid Attention Models on NPUscited by 0
  13. 2026PastForward: Faster On-Device GUI Agents via Computational Experience Reusecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Large and Small MLLMs
  2. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  3. 2024Efficient Multimodal Large Language Models: A Survey
  4. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  5. 2025SmolVLM: Redefining small and efficient multimodal models
  6. 2024On-Device Language Models: A Comprehensive Review
  7. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveys
  8. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks