mlsanityData to 7 Oct 2026

Preconditioned Visual Language Inference with Weak Supervision

arXiv 2306.01753 ↗First posted 22 May 2023cs.CL cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Humans can infer the affordance of objects by extracting related contextual preconditions for each scenario. For example, upon seeing an image of a broken cup, we can infer that this precondition prevents the cup from being used for drinking. Reasoning with preconditions of commonsense is studied in NLP where the model explicitly gets the contextual precondition. However, it is unclear if SOTA visual language models (VLMs) can extract such preconditions and infer the affordance of objects with them. In this work, we introduce the task of preconditioned visual language inference and rationalization (PVLIR). We propose a learning resource based on three strategies to retrieve weak supervision signals for the task and develop a human-verified test set for evaluation. Our results reveal the shortcomings of SOTA VLM models in the task and draw a road map to address the challenges ahead in improving them.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 53 of its 82 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

53 of 82 references.

References · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
38 more
  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  2. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  3. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  4. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  5. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  6. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
  7. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  8. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  9. 2022UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learningcited by 182
  10. 2020Adversarial Filters of Dataset Biasescited by 159
  11. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  12. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  13. 2020Captioning Images Taken by People Who Are Blindcited by 108
  14. 2020NILE : Natural Language Inference with Faithful Natural Language Explanationscited by 96
  15. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  16. 2018Ultra-Fine Entity Typingcited by 87
  17. 2021Few-Shot Self-Rationalization with Natural Language Promptscited by 83
  18. 2017HoloClean: Holistic Data Repairs with Probabilistic Inferencecited by 79
  19. 2017Neural Ranking Models with Weak Supervisioncited by 61
  20. 2022PINTO: Faithful Language Reasoning Using Prompt-Generated Rationalescited by 56
  21. 2021Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Modelcited by 39
  22. 2020Temporal Common Sense Acquisition with Minimal Supervisioncited by 38
  23. 2018Visual Entailment Task for Visually-Grounded Language Learningcited by 36
  24. 2021Cross-Modal Discrete Representation Learningcited by 35
  25. 2021Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resourcescited by 30
  26. 2022Should We Rely on Entity Mentions for Relation Extraction? Debiasing Relation Extraction with Counterfactual Analysiscited by 30
  27. 2020Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervisioncited by 27
  28. 2021"I'm Not Mad": Commonsense Implications of Negation and Contradictioncited by 26
  29. 2020Advanced Semantics for Commonsense Knowledge Extractioncited by 24
  30. 2020Beyond VQA: Generating Multi-word Answer and Rationale to Visual Questionscited by 19
  31. 2022Does Your Model Classify Entities Reasonably? Diagnosing and Mitigating Spurious Correlations in Entity Typingcited by 16
  32. 2022Extracting or Guessing? Improving Faithfulness of Event Temporal Relation Extractioncited by 15
  33. 2021AUTO-DISCERN: Autonomous Driving Using Common Sense Reasoningcited by 14
  34. 2018Grounded Textual Entailmentcited by 10
  35. 2021PaCo: Preconditions Attributed to Commonsense Knowledgecited by 9
  36. 2020Generating Rationales in Visual Question Answeringcited by 6
  37. 2016Learning from Maps: Visual Common Sense for Autonomous Drivingcited by 4
  38. 2022DU-VLG: Unifying Vision-and-Language Generation via Dual Sequence-to-Sequence Pre-trainingcited by 4

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Activating Visual Context and Commonsense Reasoning through Masked Prediction in VLMs
  2. 2026Be Faithful When Response: Returning Fluent and Grounded Answers for Vision-Language Models Reinforcement Learning
  3. 2023ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models
  4. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs
  5. 2024Probing Visual Language Priors in VLMs
  6. 2025Multi-Step Visual Reasoning with Visual Tokens Scaling and Verification
  7. 2025CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models
  8. 2026Improving Reasoning in Vision-Language Models via Perception Verified Self-Training