Ostrakon-VL: Towards Domain-Expert MLLM for Food-Service and Retail Stores
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Multimodal Large Language Models (MLLMs) have recently achieved substantial progress in general-purpose perception and reasoning. Nevertheless, their deployment in Food-Service and Retail Stores (FSRS) scenarios encounters two major obstacles: (i) real-world FSRS data, collected from heterogeneous acquisition devices, are highly noisy and lack auditable, closed-loop data curation, which impedes the construction of high-quality, controllable, and reproducible training corpora; and (ii) existing evaluation protocols do not offer a unified, fine-grained and standardized benchmark spanning single-image, multi-image, and video inputs, making it challenging to objectively gauge model robustness. To address these challenges, we first develop Ostrakon-VL, an FSRS-oriented MLLM based on Qwen3-VL-8B. Second, we introduce ShopBench, the first public benchmark for FSRS. Third, we propose QUAD (Quality-aware Unbiased Automated Data-curation), a multi-stage multimodal instruction data curation pipeline. Leveraging a multi-stage training strategy, Ostrakon-VL achieves an average score of 60.1 on ShopBench, establishing a new state of the art among open-source MLLMs with comparable parameter scales and diverse architectures. Notably, it surpasses the substantially larger Qwen3-VL-235B-A22B (59.4) by +0.7, and exceeds the same-scale Qwen3-VL-8B (55.3) by +4.8, demonstrating significantly improved parameter efficiency. These results indicate that Ostrakon-VL delivers more robust and reliable FSRS-centric perception and decision-making capabilities. To facilitate reproducible research, we will publicly release Ostrakon-VL and the ShopBench benchmark.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 67 references that are in the map. First posted on arXiv 29 Jan 2026.
Builds on
52 of 67 references.
Closest ancestors · 13
Same direction, newest first
- 2025Kwai Keye-VL Technical Report2 Jul
- 2025On Path to Multimodal Generalist: General-Level and General-Bench7 May
- 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
- 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
- 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
- 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
- 2024CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark22 Jan
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
- 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
- 2023Improved Baselines with Visual Instruction Tuning5 Oct
- 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
1 more
Other references · 39
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2023Visual Instruction Tuningcited by 7,869
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
- 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
- 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
- 2024PaliGemma: A versatile 3B VLM for transfercited by 696
27 more
- 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
- 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
- 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
- 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
- 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
- 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 401
- 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
- 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
- 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
- 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
- 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 170
- 2023CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?cited by 90
- 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
- 2025Kwai Keye-VL 1.5 Technical Reportcited by 73
- 2019AI2D-RST: A multimodal corpus of 1000 primary school science diagramscited by 60
- 2025Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoningcited by 49
- 2024Binary Classifier Optimization for Large Language Model Alignmentcited by 45
- 2024DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Expertscited by 43
- 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 42
- 2024Open-FinLLMs: Open Multimodal Large Language Models for Financial Applicationscited by 36
- 2025Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoningcited by 32
- 2024Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generationcited by 31
- 2023Jellyfish: A Large Language Model for Data Preprocessingcited by 20
- 2023VDC: Versatile Data Cleanser based on Visual-Linguistic Inconsistency by Multimodal Large Language Modelscited by 9
- 2024Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Designcited by 9
- 2025CoddLLM: Empowering Large Language Models for Data Analyticscited by 6
- 2025Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Modelcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024FoodMLLM-JP: Leveraging Multimodal Large Language Models for Japanese Recipe Generation
- 2023Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Models
- 2026What Matters for Grocery Product Retrieval with Open Source Vision Language Models
- 2025Benchmarking Large and Small MLLMs
- 2025Trust the Model: Compact VLMs as In-Context Judges for Image-Text Data Quality
- 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
- 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
- 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone