mlsanityData to 14 Oct 2026

Language as a Label: Zero-Shot Multimodal Classification of Everyday Postures under Data Scarcity

arXiv 2510.13364 ↗First posted 15 Oct 2025cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent Vision-Language Models (VLMs) enable zero-shot classification by aligning images and text in a shared space, a promising approach for data-scarce conditions. However, the influence of prompt design on recognizing visually similar categories, such as human postures, is not well understood. This study investigates how prompt specificity affects the zero-shot classification of sitting, standing, and walking/running on a small, 285-image COCO-derived dataset. A suite of modern VLMs, including OpenCLIP, MetaCLIP 2, and SigLip, were evaluated using a three-tiered prompt design that systematically increases linguistic detail. Our findings reveal a compelling, counter-intuitive trend: for the highest-performing models (MetaCLIP 2 and OpenCLIP), the simplest, most basic prompts consistently achieve the best results. Adding descriptive detail significantly degrades performance for instance, MetaCLIP 2's multi-class accuracy drops from 68.8% to 55.1% a phenomenon we term "prompt overfitting". Conversely, the lower-performing SigLip model shows improved classification on ambiguous classes when given more descriptive, body-cue-based prompts.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 26 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

20 of 26 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Image Recognition with Vision and Language Embeddings of VLMs11 Sep
  2. 2025Meta CLIP 2: A Worldwide Scaling Recipe29 Jul
  3. 2025From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection19 May
  4. 2024Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation3 Jul
  5. 2023Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity28 Jun

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  7. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  8. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  9. 2024YOLOv11: An Overview of the Key Architectural Enhancementscited by 425
  10. 2020Temporal Interlacing Networkcited by 45
  11. 2022End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillationcited by 30
  12. 2017Zero-Shot Activity Recognition with Verb Attribute Inductioncited by 15
3 more
  1. 2024Text-Enhanced Zero-Shot Action Recognition: A training-free approachcited by 6
  2. 2024Are Visual-Language Models Effective in Action Recognition? A Comparative Studycited by 4
  3. 2024MI-VisionShot: Few-shot adaptation of vision-language models for slide-level classification of histopathological imagescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TADS: Task-Aware Data Selection for Multi-Task Multimodal Pre-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Exploring Vision Language Models for Multimodal and Multilingual Stance Detection
  2. 2024Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs
  3. 2024What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models
  4. 2026Beyond Heuristic Prompting: A Concept-Guided Bayesian Framework for Zero-Shot Image Recognition
  5. 2024How Does Diverse Interpretability of Textual Prompts Impact Medical Vision-Language Zero-Shot Tasks?
  6. 2026CARPRT: Class-Aware Zero-Shot Prompt Reweighting for Black-Box Vision-Language Models
  7. 2023Pushing Boundaries: Exploring Zero Shot Object Classification with Large Multimodal Models
  8. 2026Will It Zero-Shot?: Predicting Zero-Shot Classification Performance For Arbitrary Queries