mlsanityData to 7 Oct 2026

Magma: A Foundation Model for Multimodal AI Agents

arXiv 2502.13130 ↗First posted 18 Feb 2025cs.CV cs.AI cs.HCcited by 138

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

We present Magma, a foundation model that serves multimodal AI agentic tasks in both the digital and physical worlds. Magma is a significant extension of vision-language (VL) models in that it not only retains the VL understanding ability (verbal intelligence) of the latter, but is also equipped with the ability to plan and act in the visual-spatial world (spatial-temporal intelligence) and complete agentic tasks ranging from UI navigation to robot manipulation. To endow the agentic capabilities, Magma is pretrained on large amounts of heterogeneous datasets spanning from images, videos to robotics data, where the actionable visual objects (e.g., clickable buttons in GUI) in images are labeled by Set-of-Mark (SoM) for action grounding, and the object movements (e.g., the trace of human hands or robotic arms) in videos are labeled by Trace-of-Mark (ToM) for action planning. Extensive experiments show that SoM and ToM reach great synergy and facilitate the acquisition of spatial-temporal intelligence for our Magma model, which is fundamental to a wide range of tasks as shown in Fig.1. In particular, Magma creates new state-of-the-art results on UI navigation and robotic manipulation tasks, outperforming previous models that are specifically tailored to these tasks. On image and video-related multimodal tasks, Magma also compares favorably to popular large multimodal models that are trained on much larger datasets. We make our model and code public for reproducibility at https://microsoft.github.io/Magma.

Key facts. Cited by 138 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 103 of its 130 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

103 of 130 references.

References · 103

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022A ConvNet for the 2020scited by 3,545
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  12. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  13. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  14. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  15. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  16. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  17. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  18. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  19. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  20. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  21. 2021Grounded Language-Image Pre-trainingcited by 1,147
  22. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  23. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  24. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  25. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  26. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  27. 2016Modeling Context in Referring Expressionscited by 1,054
  28. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  29. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  30. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  31. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  32. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  33. 2016A Diagram Is Worth A Dozen Imagescited by 793
  34. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  35. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  36. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  37. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  38. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692

50 most cited of 103 All 103 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 77; 2026: 61.

138 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  2. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  3. 2026World Action Models are Zero-shot Policiescited by 411
  4. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 183
  5. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  6. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
  7. 2025GR-3 Technical Reportcited by 110
  8. 2025Agentic Large Language Models, a surveycited by 105
  9. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  10. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  11. 2026A Pragmatic VLA Foundation Modelcited by 97
  12. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
38 more
  1. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  2. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
  3. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  4. 2025Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videoscited by 63
  5. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  6. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  7. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 58
  8. 2025Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directionscited by 46
  9. 2026Xiaomi-Robotics-0: An Open-Sourced Vision-Language-Action Model with Real-Time Executioncited by 42
  10. 2025ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learningcited by 41
  11. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Modelcited by 36
  12. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
  13. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  14. 2025ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Contextcited by 34
  15. 2025Don't Blind Your VLA: Aligning Visual Representations for OOD Generalizationcited by 34
  16. 20253D Aware Region Prompted Vision Language Modelcited by 31
  17. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  18. 2025EO-1: An Open Unified Embodied Foundation Model for General Robot Controlcited by 30
  19. 2026Native Video-Action Pretraining for Generalizable Robot Controlcited by 30
  20. 2026StarVLA-α: Reducing Complexity in Vision-Language-Action Systemscited by 27
  21. 2026From Foundation to Application: Improving VLA Models in Practicecited by 25
  22. 2025CronusVLA: Towards Efficient and Robust Manipulation via Multi-Frame Vision-Language-Action Modelingcited by 24
  23. 2025InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimizationcited by 24
  24. 2026LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queriescited by 24
  25. 2025Foundation Models for Spatio-Temporal Data Science: A Tutorial and Surveycited by 23
  26. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Modelscited by 23
  27. 2025Embodied AI: From LLMs to World Modelscited by 23
  28. 2025Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representationscited by 21
  29. 2026Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planningcited by 20
  30. 2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Modelcited by 19
  31. 2026VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Modelscited by 17
  32. 2025AsyncVLA: Asynchronous Flow Matching for Vision-Language-Action Modelscited by 16
  33. 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 15
  34. 2025CaptionQA: Is Your Caption as Useful as the Image Itself?cited by 14
  35. 2026FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Modelcited by 14
  36. 2026Next Forcing: Causal World Modeling with Multi-Chunk Predictioncited by 14
  37. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalizationcited by 13
  38. 2025NaviTrace: Evaluating Embodied Navigation of Vision-Language Modelscited by 13

50 most cited of 138 All 138 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Agent AI: Surveying the Horizons of Multimodal Interaction
  2. 2025DeepEyesV2: Toward Agentic Multimodal Model
  3. 2025Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks
  4. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents
  5. 2026A.D.A.M.O. (Agent for language-Driven Actions with Multimodal Observations): A Visual-Symbolic Framework for Virtual Humans
  6. 2024Octopus v3: Technical Report for On-device Sub-billion Multimodal AI Agent
  7. 2026Kimi K2.5: Visual Agentic Intelligence
  8. 2024A Survey on Vision-Language-Action Models for Embodied AI