mlsanityData to 7 Oct 2026

Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models

arXiv 2501.14818 ↗First posted 20 Jan 2025cs.CV cs.AI cs.LGcited by 69

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recently, promising progress has been made by open-source vision-language models (VLMs) in bringing their capabilities closer to those of proprietary frontier models. However, most open-source models only publish their final model weights, leaving the critical details of data strategies and implementation largely opaque. In this work, we address VLM post-training from a data-centric perspective, showing the key role of data strategy in developing frontier VLMs. By studying and building our post-training data strategy from scratch, we share detailed insights into the development processes, aiming to benefit the development of competitive models for the open-source community. Our introduced data strategy, together with training recipes and model design, leads to a family of performant VLMs named Eagle2. Specifically, Eagle2-9B achieves state-of-the-art results across various multimodal benchmarks, matching certain competitive models with up to 70B parameters.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 97 of its 226 references that are in the map. First posted on arXiv 20 Jan 2025.

Builds on

97 of 226 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024NVILA: Efficient Frontier Visual Language Models5 Dec
  2. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models25 Sep
  3. 2024Building and better understanding vision-language models: insights and future directions22 Aug
  4. 2024SPARK: Multi-Vision Sensor Perception and Reasoning Benchmark for Large-scale Vision-Language Models22 Aug
  5. 2024VILA2: VILA Augmented VILA24 Jul
  6. 2024MMRA: A Benchmark for Evaluating Multi-Granularity and Multi-Image Relational Association Capabilities in Large Visual Language Models24 Jul
  7. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine11 Jul
  8. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  9. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences16 Jun
  10. 2024What matters when building vision-language models?3 May
  11. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models27 Mar
  12. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models1 Mar
3 more
  1. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models18 Feb
  2. 2023PaLI-3 Vision Language Models: Smaller, Faster, Stronger13 Oct
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug

Other references · 82

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  10. 2024Qwen2 Technical Reportcited by 2,076
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
38 more
  1. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  5. 2023A Survey on Multimodal Large Language Modelscited by 898
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  7. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  8. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  11. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  12. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  13. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  14. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  15. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  16. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  17. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  18. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  19. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  20. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  21. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  22. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 314
  23. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  24. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  25. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  26. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  27. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  28. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  29. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 235
  30. 2024mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understandingcited by 220
  31. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  32. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  33. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  34. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  35. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  36. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  37. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  38. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuningcited by 149

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 38; 2026: 31.

69 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  3. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  4. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 152
  5. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 140
  6. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  7. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  8. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  9. 2025InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulationcited by 71
  10. 2026Being-H0.7: A Latent World-Action Model from Egocentric Videoscited by 68
  11. 2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulatorscited by 65
  12. 2026RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robotscited by 65
38 more
  1. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  2. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  3. 2025FineVision: Open Data Is All You Needcited by 51
  4. 2025Dual-Stream Diffusion for World-Model Augmented Vision-Language-Action Modelcited by 33
  5. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  6. 2025Interpretable and Testable Vision Features via Sparse Autoencoderscited by 24
  7. 2025Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Modelcited by 24
  8. 2026Adaptive Action Chunking at Inference-time for Vision-Language-Action Modelscited by 24
  9. 2025Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoningcited by 23
  10. 2025Referring to Any Personcited by 20
  11. 2025ReXVQA: A Large-scale Visual Question Answering Benchmark for Generalist Chest X-ray Understandingcited by 20
  12. 2026Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planningcited by 20
  13. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
  14. 2026Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stackcited by 20
  15. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoTcited by 16
  16. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMscited by 15
  17. 2025High-Resolution Visual Reasoning via Multi-Turn Grounding-Based Reinforcement Learningcited by 13
  18. 2025FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Modelcited by 11
  19. 2025Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Modelscited by 10
  20. 2025TwinVLA: Data-Efficient Bimanual Manipulation with Twin Single-Arm Vision-Language-Action Modelscited by 10
  21. 2025MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generationcited by 9
  22. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  23. 2025LocateAnything3D: Vision-Language 3D Detection with Chain-of-Sightcited by 8
  24. 2026Nemotron ColEmbed V2: Top-Performing Late Interaction Embedding Models for Visual Document Retrievalcited by 8
  25. 2026AgentVLN: Towards Agentic Vision-and-Language Navigationcited by 8
  26. 2026Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulationcited by 8
  27. 2026Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularizationcited by 7
  28. 2026FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulationcited by 6
  29. 2025Spatio-Temporal LLM: Reasoning about Environments and Actionscited by 5
  30. 2026Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Modelscited by 5
  31. 2025HAWAII: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Modelscited by 4
  32. 2025TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognitioncited by 4
  33. 2025RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulation Environmentscited by 3
  34. 2026SignVLA: A Gloss-Free Vision-Language-Action Framework for Real-Time Sign Language-Guided Robotic Manipulationcited by 3
  35. 2026LARA: Latent Action Representation Alignment for Vision-Language-Action Modelscited by 3
  36. 2025Beyond Success: Refining Elegant Robot Manipulation from Mixed-Quality Data via Just-in-Time Interventioncited by 2
  37. 2026VCA: Vision-Click-Action Framework for Precise Manipulation of Segmented Objects in Target Ambiguous Environmentscited by 2
  38. 2026EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-trainingcited by 2

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026DatBench: Discriminative, Faithful, and Efficient VLM Evaluations
  2. 2025Scalable Vision Language Model Training via High Quality Data Curation
  3. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Models
  4. 2026Empirical Recipes for Efficient and Compact Vision-Language Models
  5. 2025Data Metabolism: An Efficient Data Design Schema For Vision Language Model
  6. 2026JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
  7. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  8. 2025From Behavioral Performance to Internal Competence: Interpreting Vision-Language Models with VLM-Lens