mlsanityData to 14 Oct 2026

You Are What You Prompt: Prompt Quality, Domain Shift, and Uncertainty in Agrifood Vision-Language Models

arXiv 2608.18116 ↗First posted 1 Jul 2026cs.CL cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Agricultural vision-language models

Abstract (arXiv)

Vision-language models enable zero-shot classification through natural language prompts, but performance is sensitive to prompt formulation, especially in specialized domains. Zero-shot Prompt Ensembling (ZPE) addresses this by weighting prompts by discriminative signal, yet its behavior under domain shift remains unexplored. We evaluate ZPE in the agrifood domain using CLIP and SigLIP across four datasets and four prompt pools, spanning in-distribution (ID) food and out-of-distribution agricultural benchmarks. ZPE provides limited benefit under ID conditions but substantially improves performance and calibration under domain shift, where domain-specific pools of 51-52 prompts consistently outperform generic pools of 247-426. Lexical analysis shows that ZPE acts as an unsupervised domain-alignment detector without label access. We further introduce PID (Prompt-based Inconsistency Detection), which repurposes prompt disagreement as epistemic uncertainty, improving failure detection under severe domain shift where standard confidence measures collapse.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agricultural vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 23 references that are in the map. First posted on arXiv 1 Jul 2026.

Builds on

10 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025AgMMU: A Comprehensive Agricultural Multimodal Understanding Benchmark14 Apr
  2. 2024AgriCLIP: Adapting CLIP for Agriculture and Livestock via Domain-Specialized Cross-Model Alignment2 Oct

Other references · 8

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  6. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
  7. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  8. 2025WisWheat: A Three-Tiered Vision-Language Dataset for Wheat Managementcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Are vision-language models ready to zero-shot replace supervised classification models in agriculture?
  2. 2026Vision-language models know more about agriculture than they show and rubric-grounded verifications close the gap
  3. 2025AgriGPT-VL: Agricultural Vision-Language Understanding Suite
  4. 2025Self-Consistency in Vision-Language Models for Precision Agriculture: Multi-Response Consensus for Crop Disease Management
  5. 2026AgriChat: A Multimodal Large Language Model for Agriculture Image Understanding
  6. 2024AgroGPT: Efficient Agricultural Vision-Language Model with Expert Tuning
  7. 2026CropVLM: A Domain-Adapted Vision-Language Model for Open-Set Crop Analysis
  8. 2024Leveraging Vision Language Models for Specialized Agricultural Tasks