# STATUS Bench: A Rigorous Benchmark for Evaluating Object State Understanding in Vision-Language Models

arXiv: https://arxiv.org/abs/2510.22571 · HTML: https://mlsanity.com/paper/2510.22571/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 31 of its 56 references that are in the map.
- First posted on arXiv 26 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [NVILA: Efficient Frontier Visual Language Models](https://mlsanity.com/paper/2412.04468/) (2024)
- [Learning to Describe Differences Between Pairs of Similar Images](https://mlsanity.com/paper/1808.10584/) (2018)
- [Urban Change Detection for Multispectral Earth Observation Using Convolutional Neural Networks](https://mlsanity.com/paper/1810.08468/) (2018)
- [S2Looking: A Satellite Side-Looking Dataset for Building Change Detection](https://mlsanity.com/paper/2107.09244/) (2021)
- [VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution](https://mlsanity.com/paper/2306.12424/) (2023)
- [Describing and Localizing Multiple Changes with Transformers](https://mlsanity.com/paper/2103.14146/) (2021)
- [Weakly Supervised Silhouette-based Semantic Scene Change Detection](https://mlsanity.com/paper/1811.11985/) (2018)
- [Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videos](https://mlsanity.com/paper/2203.11637/) (2022)
- [Learning Object State Changes in Videos: An Open-World Perspective](https://mlsanity.com/paper/2312.11782/) (2023)
- [Joint Discovery of Object States and Manipulation Actions](https://mlsanity.com/paper/1702.02738/) (2017)
- [Chop & Learn: Recognizing and Generating Object-State Compositions](https://mlsanity.com/paper/2309.14339/) (2023)
- [Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flows](https://mlsanity.com/paper/2209.05840/) (2022)
- [ChangeNet: Multi-Temporal Asymmetric Change Detection Dataset](https://mlsanity.com/paper/2312.17428/) (2023)
- [Detecting Object States vs Detecting Objects: A New Dataset and a Quantitative Experimental Study](https://mlsanity.com/paper/2112.08281/) (2021)

## Built on by (most cited first, 3 of 3)

- [DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?](https://mlsanity.com/paper/2605.29615/) (2026)
- [Action Without Interaction: Probing the Physical Foundations of Video LMMs via Contact-Release Detection](https://mlsanity.com/paper/2511.20162/) (2025)
- [Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?](https://mlsanity.com/paper/2607.08514/) (2026)

## Abstract (arXiv)

Object state recognition aims to identify the specific condition of objects, such as their positional states (e.g., open or closed) and functional states (e.g., on or off). While recent Vision-Language Models (VLMs) are capable of performing a variety of multimodal tasks, it remains unclear how precisely they can identify object states. To alleviate this issue, we introduce the STAte and Transition UnderStanding Benchmark (STATUS Bench), the first benchmark for rigorously evaluating the ability of VLMs to understand subtle variations in object states in diverse situations. Specifically, STATUS Bench introduces a novel evaluation scheme that requires VLMs to perform three tasks simultaneously: object state identification (OSI), image retrieval (IR), and state change identification (SCI). These tasks are defined over our fully hand-crafted dataset involving image pairs, their corresponding object state descriptions and state change descriptions. Furthermore, we introduce a large-scale training dataset, namely STATUS Train, which consists of 13 million semi-automatically created descriptions. This dataset serves as the largest resource to facilitate further research in this area. In our experiments, we demonstrate that STATUS Bench enables rigorous consistency evaluation and reveal that current state-of-the-art VLMs still significantly struggle to capture subtle object state distinctions. Surprisingly, under the proposed rigorous evaluation scheme, most open-weight VLMs exhibited chance-level zero-shot performance. After fine-tuning on STATUS Train, Qwen2.5-VL achieved performance comparable to Gemini 2.0 Flash. These findings underscore the necessity of STATUS Bench and Train for advancing object state recognition in VLM research.