mlsanityData to 7 Oct 2026

ZAYA1-VL-8B Technical Report

arXiv 2605.08560 ↗First posted 8 May 2026cs.CV cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

also in 1 more
Abstract (arXiv)

We present ZAYA1-VL-8B, a compact mixture-of-experts vision-language model built upon our in-house language model, ZAYA1-8B. Despite its compact size, ZAYA1-VL achieves performance competitive with leading base models such as Molmo2-4B and InternVL3.5-4B, while surpassing models including Qwen2.5-VL-3B, PLM-3B, and MolmoE-1B across a range of image understanding, reasoning, and counting benchmarks. The architecture incorporates two key innovations: (1) vision-specific LoRA adapters integrated into the LLM to increase modality-specific capacity without increasing the number of experts, and (2) bidirectional attention over image tokens within the LLM to enhance visual understanding. We detail the full training pipeline including data composition at each stage, sequence packing, and the attention masking scheme. The model comprises 9.2B total parameters, with 1.4B active parameters including the vision encoder, and is publicly available at https://huggingface.co/Zyphra/ZAYA1-VL.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 112 of its 149 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

112 of 149 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025FineVision: Open Data Is All You Need20 Oct
  2. 2025From Pixels to Words -- Towards Native Vision-Language Primitives at Scale16 Oct
  3. 2025Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models15 Jun
  4. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models25 Sep
  5. 2024Building and better understanding vision-language models: insights and future directions22 Aug
  6. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  7. 2024What matters when building vision-language models?3 May
  8. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models1 Mar
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 101

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2025Qwen2.5-VL Technical Reportcited by 5,418
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
38 more
  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2025Qwen3-VL Technical Reportcited by 2,534
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  13. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  14. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  15. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  16. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  17. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  18. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  19. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  20. 2022Token Merging: Your ViT But Fastercited by 995
  21. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  22. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  23. 2016A Diagram Is Worth A Dozen Imagescited by 793
  24. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  25. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  26. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  27. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  28. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  29. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  30. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 527
  31. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  32. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  33. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  34. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  35. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
  36. 2016Synthetic Data for Text Localisation in Natural Imagescited by 396
  37. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  38. 2025Muon is Scalable for LLM Trainingcited by 381

50 most cited of 101 All 101 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026DataComp-VLM: Improved Open Datasets for Vision-Language Modelscited by 1
  2. 2026Zamba2-VL Technical Reportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025jina-vlm: Small Multilingual Vision Language Model
  2. 2025MoCHA: Advanced Vision-Language Reasoning with MoE Connector and Hierarchical Group Attention
  3. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Context
  4. 2024LM4LV: A Frozen Large Language Model for Low-level Vision Tasks
  5. 2023LOVM: Language-Only Vision Model Selection
  6. 2024MouSi: Poly-Visual-Expert Vision-Language Models
  7. 2024MoAI: Mixture of All Intelligence for Large Language and Vision Models
  8. 2026Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance