mlsanityData to 14 Oct 2026

Self-Evolving Visual Concept Library using Vision-Language Critics

arXiv 2504.00185 ↗First posted 31 Mar 2025cs.CV cs.LGcited by 2

On the map: Synthetic data and self-improvement for LLMs → Self-improving MLLMs and synthetic data

Abstract (arXiv)

We study the problem of building a visual concept library for visual recognition. Building effective visual concept libraries is challenging, as manual definition is labor-intensive, while relying solely on LLMs for concept generation can result in concepts that lack discriminative power or fail to account for the complex interactions between them. Our approach, ESCHER, takes a library learning perspective to iteratively discover and improve visual concepts. ESCHER uses a vision-language model (VLM) as a critic to iteratively refine the concept library, including accounting for interactions between concepts and how they affect downstream classifiers. By leveraging the in-context learning abilities of LLMs and the history of performance using various concepts, ESCHER dynamically improves its concept generation strategy based on the VLM critic's feedback. Finally, ESCHER does not require any human annotations, and is thus an automated plug-and-play framework. We empirically demonstrate the ability of ESCHER to learn a concept library for zero-shot, few-shot, and fine-tuning visual classification tasks. This work represents, to our knowledge, the first application of concept library learning to real-world visual tasks.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Self-improving MLLMs and synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 23 of its 35 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

23 of 35 references.

References · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  10. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  11. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  12. 2022Visual Classification via Description from Large Language Modelscited by 336
11 more
  1. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  2. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  3. 2017Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictionscited by 234
  4. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  5. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 134
  6. 2024Symbolic Regression with a Learned Concept Librarycited by 83
  7. 2023Learning Concise and Descriptive Attributes for Visual Recognitioncited by 73
  8. 2024Improved Zero-Shot Classification by Adapting VLMs with Text Descriptionscited by 54
  9. 2023LILO: Learning Interpretable Libraries by Compressing and Documenting Codecited by 49
  10. 2021Leveraging Language to Learn Program Abstractions and Search Heuristicscited by 45
  11. 2024Evolving Interpretable Visual Classifiers with Large Language Modelscited by 18

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025VisPlay: Self-Evolving Vision-Language Models from Imagescited by 47
  2. 2025Beyond Accuracy: Metrics that Uncover What Makes a 'Good' Visual Descriptorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Bridging the gap to real-world language-grounded visual concept learning
  2. 2024Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language Models
  3. 2025VCM: Vision Concept Modeling Based on Implicit Contrastive Learning with Vision-Language Instruction Fine-Tuning
  4. 2024Rethinking VLMs and LLMs for Image Classification
  5. 2023LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptions
  6. 2024Conceptual Codebook Learning for Vision-Language Models
  7. 2026DAIT: Distillation from Vision-Language Models to Lightweight Classifiers with Adaptive Intermediate Teacher Transfer
  8. 2024Improving Concept Alignment in Vision-Language Concept Bottleneck Models