mlsanityData to 7 Oct 2026

Valley2: Exploring Multimodal Models with Scalable Vision-Language Design

arXiv 2501.05901 ↗First posted 10 Jan 2025cs.CVcited by 20

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recently, vision-language models have made remarkable progress, demonstrating outstanding capabilities in various tasks such as image captioning and video understanding. We introduce Valley2, a novel multimodal large language model designed to enhance performance across all domains and extend the boundaries of practical applications in e-commerce and short video scenarios. Notably, Valley2 achieves state-of-the-art (SOTA) performance on e-commerce benchmarks, surpassing open-source models of similar size by a large margin (79.66 vs. 72.76). Additionally, Valley2 ranks second on the OpenCompass leaderboard among models with fewer than 10B parameters, with an impressive average score of 67.4. The code and model weights are open-sourced at https://github.com/bytedance/Valley.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 68 of its 85 references that are in the map. First posted on arXiv 10 Jan 2025.

Builds on

68 of 85 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  2. 2024BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices16 Nov
  3. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  4. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data24 Oct
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  6. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders28 Aug
  7. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Models16 Aug
  8. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  10. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Model31 May
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  12. 2024Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning18 Feb
4 more
  1. 2023Generative Multimodal Models are In-Context Learners20 Dec
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
38 more
  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  2. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  3. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  4. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  5. 2016Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Networkcited by 1,199
  6. 2016Modeling Context in Referring Expressionscited by 1,054
  7. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  8. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  9. 2016A Diagram Is Worth A Dozen Imagescited by 793
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  11. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  12. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  13. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  14. 2021OCR-free Document Understanding Transformercited by 400
  15. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  16. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 376
  17. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  18. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  19. 2019Scene Text Visual Question Answeringcited by 327
  20. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  21. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  22. 2022Visual Spatial Reasoningcited by 288
  23. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  24. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  25. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  26. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 225
  27. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  28. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  29. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 193
  30. 2021TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene textcited by 170
  31. 2018TallyQA: Answering Complex Counting Questionscited by 152
  32. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  33. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  34. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 119
  35. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 112
  36. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 111
  37. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 90
  38. 2022Syntax-Aware Network for Handwritten Mathematical Expression Recognitioncited by 68

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 14; 2026: 6.

20 papers.

Sort:

By area:

  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2025R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPOcited by 52
  3. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  4. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  5. 2025Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigationcited by 20
  6. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  7. 2025GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinkingcited by 14
  8. 2025Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Modelscited by 12
  9. 2025VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?cited by 7
  10. 2025Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMscited by 6
  11. 2025iSafetyBench: A video-language benchmark for safety in industrial environmentcited by 5
  12. 2025Benchmarking Multimodal Large Language Models for Face Recognitioncited by 5
8 more
  1. 2025Unleashing Perception-Time Scaling to Multimodal Reasoning Modelscited by 2
  2. 2025Interpretable Open-Vocabulary Referring Object Detection with Reverse Contrast Attentioncited by 1
  3. 2026Enhancing Video Representations with Spatiotemporal-Semantic Residual to Mitigate Hallucinations in Video Large Multimodal Modelscited by 1
  4. 2026Demographic Fairness in Multimodal LLMs: A Benchmark of Gender and Ethnicity Bias in Face Verificationcited by 1
  5. 2026Valley3: Scaling Omni Foundation Models for E-commercecited by 1
  6. 2026R1-SyntheticVL: Is Synthetic Data from Generative Models Ready for Multimodal Large Language Model?cited by 0
  7. 2026Learning to Deny: Action Denial in Multimodal Large Language Modelscited by 0
  8. 2026Foundation and Multimodal Large Language Models for Face Presentation and Morph Attack Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities
  2. 2024PALO: A Polyglot Large Multimodal Model for 5B People
  3. 2026Adapting Vision-Language Models for E-commerce Understanding at Scale
  4. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  5. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
  7. 2024Xmodel-VLM: A Simple Baseline for Multimodal Vision Language Model
  8. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding