mlsanityData to 14 Oct 2026

VALUED -- Vision and Logical Understanding Evaluation Dataset

arXiv 2311.12610 ↗First posted 21 Nov 2023cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Starting with early successes in computer vision tasks, deep learning based techniques have since overtaken state of the art approaches in a multitude of domains. However, it has been demonstrated time and again that these techniques fail to capture semantic context and logical constraints, instead often relying on spurious correlations to arrive at the answer. Since application of deep learning techniques to critical scenarios are dependent on adherence to domain specific constraints, several attempts have been made to address this issue. One limitation holding back a thorough exploration of this area, is a lack of suitable datasets which feature a rich set of rules. In order to address this, we present the VALUE (Vision And Logical Understanding Evaluation) Dataset, consisting of 200,000+ annotated images and an associated rule set, based on the popular board game - chess. The curated rule set considerably constrains the set of allowable predictions, and are designed to probe key semantic abilities like localization and enumeration. Alongside standard metrics, additional metrics to measure performance with regards to logical consistency is presented. We analyze several popular and state of the art vision models on this task, and show that, although their performance on standard metrics are laudable, they produce a plethora of incoherent results, indicating that this dataset presents a significant challenge for future works.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 45 references that are in the map. First posted on arXiv 21 Nov 2023.

Builds on

22 of 45 references.

References · 22

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  7. 2015Neural Module Networkscited by 376
  8. 2017A Semantic Loss Function for Deep Learning with Symbolic Knowledgecited by 270
  9. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 226
  10. 2017Learning Explanatory Rules from Noisy Datacited by 211
  11. 2020Deep Structural Causal Models for Tractable Counterfactual Inferencecited by 211
  12. 2019Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systemscited by 205
10 more
  1. 2017Neural-Symbolic Learning and Reasoning: A Survey and Interpretationcited by 196
  2. 2016Harnessing Deep Neural Networks with Logic Rulescited by 135
  3. 2017Physics-guided Neural Networks (PGNN): An Application in Lake Temperature Modelingcited by 131
  4. 2021Incorporating Domain Knowledge into Deep Neural Networkscited by 56
  5. 2023Measuring axiomatic soundness of counterfactual image modelscited by 34
  6. 2017DeepGO: Predicting protein functions from sequence and interactions using a deep ontology-aware classifiercited by 20
  7. 2020Domain Knowledge Alleviates Adversarial Attacks in Multi-Label Classifierscited by 10
  8. 2020HMIC: Hierarchical Medical Image Classification, A Deep Learning Approachcited by 7
  9. 2021Determining Chess Game State From an Imagecited by 4
  10. 2023DOST -- Domain Obedient Self-supervised Training for Multi Label Classification with Noisy Labelscited by 1

Built on by

Not cited yet.