mlsanityData to 14 Oct 2026

Ostrakon-VL: Towards Domain-Expert MLLM for Food-Service and Retail Stores

arXiv 2601.21342 ↗First posted 29 Jan 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have recently achieved substantial progress in general-purpose perception and reasoning. Nevertheless, their deployment in Food-Service and Retail Stores (FSRS) scenarios encounters two major obstacles: (i) real-world FSRS data, collected from heterogeneous acquisition devices, are highly noisy and lack auditable, closed-loop data curation, which impedes the construction of high-quality, controllable, and reproducible training corpora; and (ii) existing evaluation protocols do not offer a unified, fine-grained and standardized benchmark spanning single-image, multi-image, and video inputs, making it challenging to objectively gauge model robustness. To address these challenges, we first develop Ostrakon-VL, an FSRS-oriented MLLM based on Qwen3-VL-8B. Second, we introduce ShopBench, the first public benchmark for FSRS. Third, we propose QUAD (Quality-aware Unbiased Automated Data-curation), a multi-stage multimodal instruction data curation pipeline. Leveraging a multi-stage training strategy, Ostrakon-VL achieves an average score of 60.1 on ShopBench, establishing a new state of the art among open-source MLLMs with comparable parameter scales and diverse architectures. Notably, it surpasses the substantially larger Qwen3-VL-235B-A22B (59.4) by +0.7, and exceeds the same-scale Qwen3-VL-8B (55.3) by +4.8, demonstrating significantly improved parameter efficiency. These results indicate that Ostrakon-VL delivers more robust and reliable FSRS-centric perception and decision-making capabilities. To facilitate reproducible research, we will publicly release Ostrakon-VL and the ShopBench benchmark.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 67 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

52 of 67 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Kwai Keye-VL Technical Report2 Jul
  2. 2025On Path to Multimodal Generalist: General-Level and General-Bench7 May
  3. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  4. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  5. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  6. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  7. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  8. 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark22 Jan
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  11. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
1 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 39

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  9. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  12. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
27 more
  1. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  2. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  3. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  4. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  5. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
  6. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
  7. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  8. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  9. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  10. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
  11. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 170
  12. 2023CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?cited by 90
  13. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  14. 2025Kwai Keye-VL 1.5 Technical Reportcited by 73
  15. 2019AI2D-RST: A multimodal corpus of 1000 primary school science diagramscited by 60
  16. 2025Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoningcited by 49
  17. 2024Binary Classifier Optimization for Large Language Model Alignmentcited by 45
  18. 2024DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Expertscited by 43
  19. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 42
  20. 2024Open-FinLLMs: Open Multimodal Large Language Models for Financial Applicationscited by 36
  21. 2025Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoningcited by 32
  22. 2024Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generationcited by 31
  23. 2023Jellyfish: A Large Language Model for Data Preprocessingcited by 20
  24. 2023VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Modelscited by 9
  25. 2024Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Designcited by 9
  26. 2025CoddLLM: Empowering Large Language Models for Data Analyticscited by 6
  27. 2025Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Modelcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generation
  2. 2023Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Models
  3. 2026What Matters for Grocery Product Retrieval with Open Source Vision Language Models
  4. 2025Benchmarking Large and Small MLLMs
  5. 2025Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
  6. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  7. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  8. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone