# Language as a Label: Zero-Shot Multimodal Classification of Everyday Postures under Data Scarcity

arXiv: https://arxiv.org/abs/2510.13364 · HTML: https://mlsanity.com/paper/2510.13364/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 15 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 20 of 20)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [YOLOv11: An Overview of the Key Architectural Enhancements](https://mlsanity.com/paper/2410.17725/) (2024)
- [Meta CLIP 2: A Worldwide Scaling Recipe](https://mlsanity.com/paper/2507.22062/) (2025)
- [Temporal Interlacing Network](https://mlsanity.com/paper/2001.06499/) (2020)
- [End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2204.03541/) (2022)
- [Improving Zero-shot Generalization of Learned Prompts via Unsupervised Knowledge Distillation](https://mlsanity.com/paper/2407.03056/) (2024)
- [Zero-Shot Activity Recognition with Verb Attribute Induction](https://mlsanity.com/paper/1707.09468/) (2017)
- [Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity](https://mlsanity.com/paper/2306.16048/) (2023)
- [Text-Enhanced Zero-Shot Action Recognition: A training-free approach](https://mlsanity.com/paper/2408.16412/) (2024)
- [From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection](https://mlsanity.com/paper/2505.13233/) (2025)
- [Are Visual-Language Models Effective in Action Recognition? A Comparative Study](https://mlsanity.com/paper/2410.17149/) (2024)
- [Image Recognition with Vision and Language Embeddings of VLMs](https://mlsanity.com/paper/2509.09311/) (2025)
- [MI-VisionShot: Few-shot adaptation of vision-language models for slide-level classification of histopathological images](https://mlsanity.com/paper/2410.15881/) (2024)

## Built on by (most cited first, 1 of 1)

- [TADS: Task-Aware Data Selection for Multi-Task Multimodal Pre-Training](https://mlsanity.com/paper/2602.05251/) (2026)

## Abstract (arXiv)

Recent Vision-Language Models (VLMs) enable zero-shot classification by aligning images and text in a shared space, a promising approach for data-scarce conditions. However, the influence of prompt design on recognizing visually similar categories, such as human postures, is not well understood. This study investigates how prompt specificity affects the zero-shot classification of sitting, standing, and walking/running on a small, 285-image COCO-derived dataset. A suite of modern VLMs, including OpenCLIP, MetaCLIP 2, and SigLip, were evaluated using a three-tiered prompt design that systematically increases linguistic detail. Our findings reveal a compelling, counter-intuitive trend: for the highest-performing models (MetaCLIP 2 and OpenCLIP), the simplest, most basic prompts consistently achieve the best results. Adding descriptive detail significantly degrades performance for instance, MetaCLIP 2's multi-class accuracy drops from 68.8% to 55.1% a phenomenon we term "prompt overfitting". Conversely, the lower-performing SigLip model shows improved classification on ambiguous classes when given more descriptive, body-cue-based prompts.