mlsanityData to 14 Oct 2026

Seeing the Undefined: Chain-of-Action for Generative Semantic Labels

arXiv 2411.17406 ↗First posted 26 Nov 2024cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Recent advances in vision-language models (VLMs) have demonstrated remarkable capabilities in image classification by leveraging predefined sets of labels to construct text prompts for zero-shot reasoning. However, these approaches face significant limitations in undefined domains, where the label space is vocabulary-unknown and composite. We thus introduce Generative Semantic Labels (GSLs), a novel task that aims to predict a comprehensive set of semantic labels for an image without being constrained by a predefined labels set. Unlike traditional zero-shot classification, GSLs generates multiple semantic-level labels, encompassing objects, scenes, attributes, and relationships, thereby providing a richer and more accurate representation of image content. In this paper, we propose Chain-of-Action (CoA), an innovative method designed to tackle the GSLs task. CoA is motivated by the observation that enriched contextual information significantly improves generative performance during inference. Specifically, CoA decomposes the GSLs task into a sequence of detailed actions. Each action extracts and merges key information from the previous step, passing enriched context to the next, ultimately guiding the VLM to generate comprehensive and accurate semantic labels. We evaluate the effectiveness of CoA through extensive experiments on widely-used benchmark datasets. The results demonstrate significant improvements across key performance metrics, validating the capability of CoA to generate accurate and contextually rich semantic labels. Our work not only advances the state-of-the-art in generative semantic labels but also opens new avenues for applying VLMs in open-ended and dynamic real-world scenarios.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 40 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

23 of 40 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  11. 2015VQA: Visual Question Answeringcited by 2,506
  12. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 886
11 more
  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  2. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  3. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  4. 2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 128
  5. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  6. 2023OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examplescited by 80
  7. 2023T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answeringcited by 60
  8. 2023Learning Customized Visual Models with Retrieval-Augmented Knowledgecited by 58
  9. 2023Beyond Chain-of-Thought, Effective Graph-of-Thought Reasoning in Language Modelscited by 52
  10. 2023Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thoughtcited by 41
  11. 2024Customizing Language Model Responses with Contrastive In-Context Learningcited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026G2D: Generative-to-Discriminative Collaborative Inference for Zero-Shot Image Classification
  2. 2022Label Semantic Knowledge Distillation for Unbiased Scene Graph Generation
  3. 2024Learning Semantic-Aware Representation in Visual-Language Models for Multi-Label Recognition with Partial Labels
  4. 2024PVLR: Prompt-driven Visual-Linguistic Representation Learning for Multi-Label Image Recognition
  5. 2023Text as Image: Learning Transferable Adapter for Multi-Label Classification
  6. 2024Query-Based Knowledge Sharing for Open-Vocabulary Multi-Label Classification
  7. 2024Efficient and Context-Aware Label Propagation for Zero-/Few-Shot Training-Free Adaptation of Vision-Language Model
  8. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Models