mlsanityData to 7 Oct 2026

Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs

arXiv 2503.01743 ↗First posted 3 Mar 2025cs.CL cs.AI cs.LGcited by 491

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable language and multimodal models. Phi-4-Mini is a 3.8-billion-parameter language model trained on high-quality web and synthetic data, significantly outperforming recent open-source models of similar size and matching the performance of models twice its size on math and coding tasks requiring complex reasoning. This achievement is driven by a carefully curated synthetic data recipe emphasizing high-quality math and coding datasets. Compared to its predecessor, Phi-3.5-Mini, Phi-4-Mini features an expanded vocabulary size of 200K tokens to better support multilingual applications, as well as group query attention for more efficient long-sequence generation. Phi-4-Multimodal is a multimodal model that integrates text, vision, and speech/audio input modalities into a single model. Its novel modality extension approach leverages LoRA adapters and modality-specific routers to allow multiple inference modes combining various modalities without interference. For example, it now ranks first in the OpenASR leaderboard to date, although the LoRA component of the speech/audio modality has just 460 million parameters. Phi-4-Multimodal supports scenarios involving (vision + language), (vision + speech), and (speech/audio) inputs, outperforming larger vision-language and speech-language models on a wide range of tasks. Additionally, we experiment to further train Phi-4-Mini to enhance its reasoning capabilities. Despite its compact 3.8-billion-parameter size, this experimental version achieves reasoning performance on par with or surpassing significantly larger models, including DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B.

Key facts. Cited by 491 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 111 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

46 of 111 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  3. 2024NVLM: Open Frontier-Class Multimodal LLMs17 Sep
  4. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  6. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  7. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model29 Jan

Other references · 39

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2024GPT-4o System Cardcited by 4,467
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
27 more
  1. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  7. 2025s1: Simple test-time scalingcited by 1,311
  8. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  9. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  10. 2024Phi-4 Technical Reportcited by 675
  11. 2024Qwen2-Audio Technical Reportcited by 543
  12. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  13. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  14. 2025LIMO: Less is More for Reasoningcited by 488
  15. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  16. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  17. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 281
  18. 2024GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbotcited by 218
  19. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  20. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  21. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 124
  22. 2024Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilitiescited by 104
  23. 2024Red Teaming Visual Language Modelscited by 62
  24. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  25. 2019SocialIQA: Commonsense Reasoning about Social Interactionscited by 33
  26. 2024Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cyclecited by 32
  27. 2024LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 229; 2026: 262.

491 papers.

Sort:

By area:

  1. 2025Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulationcited by 129
  2. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 116
  3. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 101
  4. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 101
  5. 2025Paper2Poster: Towards Multimodal Poster Automation from Scientific Paperscited by 56
  6. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 53
  7. 2025OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLMcited by 53
  8. 2025WebInject: Prompt Injection Attack to Web Agentscited by 48
  9. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 47
  10. 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Mathcited by 44
  11. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 44
  12. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43
38 more
  1. 2025Distilling LLM Agent into Small Models with Retrieval and Code Toolscited by 41
  2. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 37
  3. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 35
  4. 2025VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Searchcited by 34
  5. 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 34
  6. 2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
  7. 2025SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoningcited by 32
  8. 2025Language Models are Injective and Hence Invertiblecited by 31
  9. 2025SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligencecited by 30
  10. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  11. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  12. 2026ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learningcited by 28
  13. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuningcited by 27
  14. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  15. 2025ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Modelscited by 26
  16. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 24
  17. 2025Streaming Sequence-to-Sequence Learning with Delayed Streams Modelingcited by 24
  18. 2025From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaningcited by 23
  19. 2025MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videoscited by 22
  20. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 20
  21. 2025AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMscited by 19
  22. 2025See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Modelscited by 19
  23. 2025NeuroStrike: Neuron-Level Attacks on Aligned LLMscited by 18
  24. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interactioncited by 18
  25. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 17
  26. 2025SpeechJudge: Towards Human-Level Judgment for Speech Naturalnesscited by 15
  27. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musiccited by 15
  28. 2025SafeDriveRAG: Towards Safe Autonomous Driving with Knowledge Graph-based Retrieval-Augmented Generationcited by 14
  29. 2025CVBench: Benchmarking Cross-Video Synergies for Complex Multimodal Reasoningcited by 14
  30. 2026TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Modelscited by 14
  31. 2025Unveiling the Compositional Ability Gap in Vision-Language Reasoning Modelcited by 13
  32. 2025VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understandingcited by 13
  33. 2025RSCC: A Large-Scale Remote Sensing Change Caption Dataset for Disaster Eventscited by 13
  34. 2025VeriContaminated: Assessing LLM-Driven Verilog Coding for Data Contaminationcited by 12
  35. 2025Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problemscited by 12
  36. 2025AbGen: Evaluating Large Language Models in Ablation Study Design and Evaluation for Scientific Researchcited by 12
  37. 2026WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Conceptscited by 12
  38. 2025GeoUni: A Unified Model for Generating Geometry Diagrams, Problems and Problem Solutionscited by 11

50 most cited of 491 All 491 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models
  2. 2025Benchmarking Large and Small MLLMs
  3. 2026M-MiniGPT4: Multilingual VLLM Alignment via Translated Data
  4. 2024Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency
  5. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
  6. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  8. 2023A Survey on Multimodal Large Language Models