mlsanityData to 7 Oct 2026

IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truth

arXiv 2505.22305 ↗First posted 28 May 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We present IKIWISI ("I Know It When I See It"), an interactive visual pattern generator for assessing vision-language models in video object recognition when ground truth is unavailable. IKIWISI transforms model outputs into a binary heatmap where green cells indicate object presence and red cells indicate object absence. This visualization leverages humans' innate pattern recognition abilities to evaluate model reliability. IKIWISI introduces "spy objects": adversarial instances users know are absent, to discern models hallucinating on nonexistent items. The tool functions as a cognitive audit mechanism, surfacing mismatches between human and machine perception by visualizing where models diverge from human understanding. Our study with 15 participants found that users considered IKIWISI easy to use, made assessments that correlated with objective metrics when available, and reached informed conclusions by examining only a small fraction of heatmap cells. This approach not only complements traditional evaluation methods through visual assessment of model behavior with custom object sets, but also reveals opportunities for improving alignment between human perception and machine understanding in vision-language systems.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 99 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

32 of 99 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Beyond Visual Perception: Insights from Smartphone Interaction of Visually Impaired Users with Large Multimodal Models22 Feb
  2. 2024Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Design11 Jul

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  5. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  6. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  7. 2020Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlationcited by 420
  8. 2022Visual Spatial Reasoningcited by 288
  9. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 245
  10. 2023Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Drivingcited by 231
  11. 2018Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontierscited by 117
  12. 2019GS3D: An Efficient 3D Object Detection Framework for Autonomous Drivingcited by 106
18 more
  1. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  2. 2019Do Neural Language Representations Learn Physical Commonsense?cited by 62
  3. 2022LAVIS: A Library for Language-Vision Intelligencecited by 55
  4. 2018Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Modelscited by 50
  5. 2021Towards General Purpose Vision Systemscited by 46
  6. 2023CLAIR: Evaluating Image Captions with Large Language Modelscited by 43
  7. 2019Human-Centered Artificial Intelligence and Machine Learningcited by 37
  8. 2017Towards Better Analysis of Machine Learning Models: A Visual Analytics Perspectivecited by 36
  9. 2019ConfusionFlow: A model-agnostic visualization for temporal analysis of classifier confusioncited by 15
  10. 2024Enhancing the Travel Experience for People with Visual Impairments through Multimodal Interaction: NaviGPT, A Real-Time AI-Driven Mobile Navigation Systemcited by 13
  11. 2023ARGUS: Visualization of AI-Assisted Task Guidance in ARcited by 11
  12. 2025Logic-RAG: Augmenting Large Multimodal Models with Visual-Spatial Knowledge for Road Scene Understandingcited by 6
  13. 2021Object Detection and Pose Estimation from RGB and Depth Data for Real-time, Adaptive Robotic Graspingcited by 5
  14. 2023Who's Thinking? A Push for Human-Centered Evaluation of LLMs using the XAI Playbookcited by 5
  15. 2024Identifying Crucial Objects in Blind and Low-Vision Individuals' Navigationcited by 5
  16. 2024A Dataset for Crucial Object Recognition in Blind and Low-Vision Individuals' Navigationcited by 4
  17. 2024Wheeler: A Three-Wheeled Input Device for Usable, Efficient, and Versatile Non-Visual Interactioncited by 1
  18. 2024Demonstration of Wheeler: A Three-Wheeled Input Device for Usable, Efficient, and Versatile Non-Visual Interactioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  2. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
  3. 2025Vision Language Models: A Survey of 26K Papers
  4. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
  5. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
  6. 2026The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence
  7. 2026Separating perception from reasoning in vision-language models: a model-free render ceiling for crystal structures
  8. 2025SemVink: Advancing VLMs' Semantic Understanding of Optical Illusions via Visual Global Thinking