mlsanityData to 7 Oct 2026

VisionPsy-Nano: Improving Accuracy, Efficiency, and Reliability in On-Device Vision-Language Models

arXiv 2609.31746 ↗First posted 23 Sep 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness also fits: VLM benchmarks and visual reasoning

Abstract (arXiv)

Sub-billion-parameter Vision-Language Models are increasingly viable for on-device deployment, yet compact model size alone does not guarantee usability. On a phone, such a model can still require more than two minutes to produce its first token. On-device usability depends on three axes: accuracy, efficiency, and behavioral reliability; standard benchmarks miss the third, with answers too short to expose doom loops and prompts too benign to probe adversarial safety. We introduce a diagnosis-driven post-training recipe in which a teacher VLM stress-tests the student, uncovers failure modes beyond human priors, and converts them into targeted supervision and preference alignment, supplementing generic data scaling with failure-driven optimization. Coupled with two visual-token policies, the recipe yields two accuracy-efficiency variants with improved behavioral reliability. NanoFull attains a 62.3 normalized average over 17 benchmarks, the highest among openly released ~0.5B models, +7.4 over its base at identical architecture and token budget, with doom-loop rates at or below the strongest baseline's. FlashFull retains 61.4 while cutting warm time-to-first-token on a Pixel 9 from 138,s to 6.1,s (23×). By jointly addressing all three axes, we move compact VLMs toward practical on-device usability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 58 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

51 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs4 Jan
  2. 2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback22 Mar
  3. 2024Red Teaming Visual Language Models23 Jan

Other references · 48

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2019Towards VQA Models That Can Readcited by 1,696
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2025Gemma 3 Technical Reportcited by 1,686
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
36 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  4. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  5. 2022Editing Models with Task Arithmeticcited by 1,076
  6. 2018Experience Replay for Continual Learningcited by 1,074
  7. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  8. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  9. 2016A Diagram Is Worth A Dozen Imagescited by 793
  10. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  11. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  12. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  13. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  14. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
  15. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  16. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  17. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  18. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  19. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Modelscited by 133
  20. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  21. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  22. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Modelscited by 89
  23. 2024FastVLM: Efficient Vision Encoding for Vision Language Modelscited by 76
  24. 2025Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generationcited by 54
  25. 2024TinyChart: Efficient Chart Understanding with Visual Token Merging and Program-of-Thoughts Learningcited by 51
  26. 2025FineVision: Open Data Is All You Needcited by 51
  27. 2025PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Modelscited by 49
  28. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 41
  29. 2025LFM2 Technical Reportcited by 31
  30. 2025OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Mergingcited by 12
  31. 2026Vero: An Open RL Recipe for General Visual Reasoningcited by 10
  32. 2025Delta-LLaVA: Base-then-Specialize Alignment for Token-Efficient Vision-Language Modelscited by 7
  33. 2024One VLM to Keep it Learning: Generation and Balancing for Data-free Continual Visual Question Answeringcited by 6
  34. 2026InfoSFT: Learn More and Forget Less with Information-Aware Token Weightingcited by 3
  35. 2024Error-driven Data-efficient Large Multimodal Model Tuningcited by 2
  36. 2026Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvementcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024OmniVLM: A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inference
  2. 2024What matters when building vision-language models?
  3. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
  4. 2026StepX-Edge: An On-Device UI Vision-Language Model via Architecture-Training-Deployment Co-Design
  5. 2026Habilis-β: A Fast-Motion and Long-Lasting On-Device Vision-Language-Action Model
  6. 2026Empirical Recipes for Efficient and Compact Vision-Language Models
  7. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models
  8. 2026A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5