mlsanityData to 7 Oct 2026

How to Determine the Preferred Image Distribution of a Black-Box Vision-Language Model?

arXiv 2409.02253 ↗First posted 3 Sep 2024cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large foundation models have revolutionized the field, yet challenges remain in optimizing multi-modal models for specialized visual tasks. We propose a novel, generalizable methodology to identify preferred image distributions for black-box Vision-Language Models (VLMs) by measuring output consistency across varied input prompts. Applying this to different rendering types of 3D objects, we demonstrate its efficacy across various domains requiring precise interpretation of complex structures, with a focus on Computer-Aided Design (CAD) as an exemplar field. We further refine VLM outputs using in-context learning with human feedback, significantly enhancing explanation quality. To address the lack of benchmarks in specialized domains, we introduce CAD-VQA, a new dataset for evaluating VLMs on CAD-related visual question answering tasks. Our evaluation of state-of-the-art VLMs on CAD-VQA establishes baseline performance levels, providing a framework for advancing VLM capabilities in complex visual reasoning tasks across various fields requiring expert-level visual interpretation. We release the dataset and evaluation codes at urlhttps://github.com/asgsaeid/cadvqa.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 34 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

28 of 34 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2017On Calibration of Modern Neural Networkscited by 4,860
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2017Deep reinforcement learning from human preferencescited by 3,846
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
16 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  3. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  8. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  9. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  10. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  11. 2018ABC: A Big CAD Model Dataset For Geometric Deep Learningcited by 339
  12. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  13. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  14. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  15. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
  16. 2019Interactive Language Learning by Question Answeringcited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Probing Visual Language Priors in VLMs
  2. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
  3. 2025Visual CoT Makes VLMs Smarter but More Fragile
  4. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  5. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  6. 2026VRIQ: Benchmarking and Analyzing Visual-Reasoning IQ of VLMs
  7. 2024MM-R3: On (In-)Consistency of Vision-Language Models (VLMs)
  8. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges