# Hob-VL: A Benchmark for Visually Grounded Boolean Reasoning

arXiv: https://arxiv.org/abs/2610.01605 · HTML: https://mlsanity.com/paper/2610.01605/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 1 Oct 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 31)

- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [Chain of Thought Empowers Transformers to Solve Inherently Serial Problems](https://mlsanity.com/paper/2402.12875/) (2024)
- [LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts](https://mlsanity.com/paper/2407.04973/) (2024)
- [FOLIO: Natural Language Reasoning with First-Order Logic](https://mlsanity.com/paper/2209.00840/) (2022)
- [Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Models](https://mlsanity.com/paper/2402.14848/) (2024)
- [Cycle-Consistency for Robust Visual Question Answering](https://mlsanity.com/paper/1902.05660/) (2019)
- [Premise Order Matters in Reasoning with Large Language Models](https://mlsanity.com/paper/2402.08939/) (2024)
- [LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models](https://mlsanity.com/paper/2404.15522/) (2024)
- [CLOSURE: Assessing Systematic Generalization of CLEVR Models](https://mlsanity.com/paper/1912.05783/) (2019)
- [EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images](https://mlsanity.com/paper/2310.18652/) (2023)
- [VQA-LOL: Visual Question Answering under the Lens of Logic](https://mlsanity.com/paper/2002.08325/) (2020)
- [Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation data](https://mlsanity.com/paper/2401.17600/) (2024)
- [SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulas](https://mlsanity.com/paper/2505.14615/) (2025)
- [RobustLR: Evaluating Robustness to Logical Perturbation in Deductive Reasoning](https://mlsanity.com/paper/2205.12598/) (2022)
- [Unlocking large-scale crop field delineation in smallholder farming systems with transfer learning and weak supervision](https://mlsanity.com/paper/2201.04771/) (2022)
- [Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis](https://mlsanity.com/paper/2409.00106/) (2024)
- [EQUI-VOCAL: Synthesizing Queries for Compositional Video Events from Limited User Interactions [Technical Report]](https://mlsanity.com/paper/2301.00929/) (2023)
- [Logic Unseen: Revealing the Logical Blindspots of Vision-Language Models](https://mlsanity.com/paper/2508.11317/) (2025)
- [MuSLR: Multimodal Symbolic Logical Reasoning](https://mlsanity.com/paper/2509.25851/) (2025)
- [Combining Deep Learning and Street View Imagery to Map Smallholder Crop Types](https://mlsanity.com/paper/2309.05930/) (2023)
- [FOL-Traces: Verified First-Order Logic Reasoning Traces at Scale](https://mlsanity.com/paper/2505.14932/) (2025)
- [Continuous Latent Contexts Enable Efficient Online Learning in Transformers](https://mlsanity.com/paper/2605.09867/) (2026)
- [MM-CondChain: A Programmatically Verified Benchmark for Visually Grounded Deep Compositional Reasoning](https://mlsanity.com/paper/2603.12266/) (2026)
- [Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models](https://mlsanity.com/paper/2607.23052/) (2026)
- [Controlled Reformulation Testing for Logical Consistency in Large Language Models](https://mlsanity.com/paper/2607.14528/) (2026)

## Abstract (arXiv)

Reliable visual reasoning requires composing multiple visual observations and returning consistent answers to logically equivalent questions. We introduce Hob-VL, a benchmark for visually grounded Boolean reasoning. Hob-VL comprises two tasks: (1) evaluating whether a Boolean rule holds in an image, and (2) identifying the (unique) object satisfying a Boolean description. Hob-VL contains 6,000 human-verified balanced Yes/No questions, each defined by a Boolean combination of ten visual statements, across 1,000 generated scenes and 46 diverse labeled photographs, along with 1,000 object-identification questions over the same photographs. Our question families are deliberately constructed to challenge reasoning through misleading local cues and nested logical operations, and include symbolic and structured natural-language presentations. Across eight model configurations with thinking disabled or minimized, Boolean accuracy ranges from 48.52% to 50.57%, while the identification accuracy reaches at most 43.0%. A thinking-enabled GLM configuration achieves uneven gains while retaining substantial errors and inconsistencies. Hob-VL exposes these failures through executable reference answers and matched evaluations.