mlsanityData to 7 Oct 2026

NL-Eye: Abductive NLI for Images

arXiv 2410.02613 ↗First posted 3 Oct 2024cs.CV cs.AI cs.CLcited by 5

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Will a Visual Language Model (VLM)-based bot warn us about slipping if it detects a wet floor? Recent VLMs have demonstrated impressive capabilities, yet their ability to infer outcomes and causes remains underexplored. To address this, we introduce NL-Eye, a benchmark designed to assess VLMs' visual abductive reasoning skills. NL-Eye adapts the abductive Natural Language Inference (NLI) task to the visual domain, requiring models to evaluate the plausibility of hypothesis images based on a premise image and explain their decisions. NL-Eye consists of 350 carefully curated triplet examples (1,050 images) spanning diverse reasoning categories: physical, functional, logical, emotional, cultural, and social. The data curation process involved two steps - writing textual descriptions and generating images using text-to-image models, both requiring substantial human involvement to ensure high-quality and challenging scenes. Our experiments show that VLMs struggle significantly on NL-Eye, often performing at random baseline levels, while humans excel in both plausibility prediction and explanation quality. This demonstrates a deficiency in the abductive reasoning capabilities of modern VLMs. NL-Eye represents a crucial step toward developing VLMs capable of robust multimodal reasoning for real-world applications, including accident-prevention bots and generated video verification.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 63 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

50 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models15 Jul

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  11. 2015VQA: Visual Question Answeringcited by 2,489
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
37 more
  1. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  2. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  4. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  5. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  6. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  7. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  8. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  9. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  10. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  11. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  12. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  13. 2019Abductive Commonsense Reasoningcited by 300
  14. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 284
  15. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  16. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  17. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  18. 2024Large Language Models Meet NLP: A Surveycited by 102
  19. 2022Benchmarking Spatial Relationships in Text-to-Image Generationcited by 96
  20. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 84
  21. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  22. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  23. 2024Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphscited by 72
  24. 2024ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigationcited by 67
  25. 2024Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiologycited by 45
  26. 2023Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactualscited by 44
  27. 2022The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoningcited by 38
  28. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language modelscited by 35
  29. 2024Question Aware Vision Transformer for Multimodal Reasoningcited by 32
  30. 2023VideoCon: Robust Video-Language Alignment via Contrast Captionscited by 29
  31. 2024Diffusion Lens: Interpreting Text Encoders in Text-to-Image Pipelinescited by 28
  32. 2022There is a Time and Place for Reasoning Beyond the Imagecited by 22
  33. 2024The Colorful Future of LLMs: Evaluating and Improving LLMs as Emotional Supporters for Queer Youthcited by 20
  34. 2023Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignmentcited by 18
  35. 2023Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Modelscited by 16
  36. 2024Visual Riddles: a Commonsense and World Knowledge Challenge for Large Vision and Language Modelscited by 10
  37. 2022Evaluating a Synthetic Image Dataset Generated with Stable Diffusioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2024Physics Context Builders: A Modular Framework for Physical Reasoning in Vision-Language Modelscited by 8
  2. 2025Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Modelscited by 4
  3. 2025DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Modelscited by 2
  4. 2025Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Taskscited by 1
  5. 2026Multimodal UNcommonsense: From Odd to Ordinary and Ordinary to Oddcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AbductiveMLLM: Boosting Visual Abductive Reasoning Within MLLMs
  2. 2024VL-GLUE: A Suite of Fundamental yet Challenging Visuo-Linguistic Reasoning Tasks
  3. 2026Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
  4. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models
  5. 2025DixitWorld: Evaluating Multimodal Abductive Reasoning in Vision-Language Models with Multi-Agent Dixit Gameplay
  6. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs
  7. 2026VRIQ: Benchmarking and Analyzing Visual-Reasoning IQ of VLMs
  8. 2022Visual Abductive Reasoning