mlsanityData to 14 Oct 2026

Learning Deep Representations of Fine-grained Visual Descriptions

arXiv 1605.05395 ↗First posted 17 May 2016cs.CVcited by 182

Not on the map. Why?

Abstract (arXiv)

State-of-the-art methods for zero-shot visual recognition formulate learning as a joint embedding problem of images and side information. In these formulations the current best complement to visual features are attributes: manually encoded vectors describing shared characteristics among categories. Despite good performance, attributes have limitations: (1) finer-grained recognition requires commensurately more attributes, and (2) attributes do not provide a natural language interface. We propose to overcome these limitations by training neural language models from scratch; i.e. without pre-training and only consuming words and characters. Our proposed models train end-to-end to align with the fine-grained and category-specific content of images. Natural language provides a flexible and compact way of encoding only the salient visual aspects for distinguishing categories. By training on raw text, our model can do inference on raw text as well, providing humans a familiar mode both for annotation and retrieval. Our model achieves strong performance on zero-shot text-based image retrieval and significantly outperforms the attribute-based state-of-the-art for zero-shot classification on the Caltech UCSD Birds 200-2011 dataset.

Key facts. Cited by 182 papers in the map (data through 14 Oct 2026). First posted on arXiv 17 May 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 55; 2022: 39; 2023: 35; 2024: 23; 2025: 19; 2026: 11.

182 papers.

Sort:

By area:

  1. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  2. 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 267
  3. 2022DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detectioncited by 176
  4. 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 163
  5. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  6. 2021Probabilistic Embeddings for Cross-Modal Retrievalcited by 151
  7. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  8. 2021Fine-Grained Image Analysis with Deep Learning: A Surveycited by 135
  9. 2021Learning Graph Embeddings for Compositional Zero-shot Learningcited by 112
  10. 2021DSSL: Deep Surroundings-person Separation Learning for Text-based Person Retrievalcited by 102
  11. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 101
  12. 2021Contrastive Embedding for Generalized Zero-Shot Learningcited by 94
38 more
  1. 2022K-LITE: Learning Transferable Visual Models with External Knowledgecited by 79
  2. 2023Towards Unified Text-based Person Retrieval: A Large-scale Multi-Attribute and Language Search Benchmarkcited by 75
  3. 2021FREE: Feature Refinement for Generalized Zero-Shot Learningcited by 67
  4. 2023Open-Set Image Tagging with Multi-Grained Text Supervisioncited by 67
  5. 2022See Finer, See More: Implicit Modality Alignment for Text-based Person Retrievalcited by 65
  6. 2021HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learningcited by 57
  7. 2021Semantics Disentangling for Generalized Zero-Shot Learningcited by 56
  8. 2021Subspace Regularizers for Few-Shot Class Incremental Learningcited by 56
  9. 2021TransZero: Attribute-guided Transformer for Zero-Shot Learningcited by 54
  10. 2023RaSa: Relation and Sensitivity Aware Representation Learning for Text-based Person Searchcited by 54
  11. 2024Improved Zero-Shot Classification by Adapting VLMs with Text Descriptionscited by 54
  12. 2021Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentationcited by 53
  13. 2021Adversarial Text-to-Image Synthesis: A Reviewcited by 52
  14. 2021Learning Graph Embeddings for Open World Compositional Zero-Shot Learningcited by 49
  15. 2021Emergent Communication of Generalizationscited by 42
  16. 2021Knowledge-aware Zero-Shot Learning: Survey and Perspectivecited by 41
  17. 2022CAIBC: Capturing All-round Information Beyond Color for Text-based Person Retrievalcited by 39
  18. 2022Look Before You Leap: Improving Text-based Person Retrieval by Learning A Consistent Cross-modal Common Manifoldcited by 38
  19. 2022Mutual Information Divergence: A Unified Metric for Multimodal Generative Modelscited by 35
  20. 2024TaxaBind: A Unified Embedding Space for Ecological Applicationscited by 35
  21. 2022StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesiscited by 33
  22. 2022Grounding Visual Representations with Texts for Domain Generalizationcited by 30
  23. 2024AutoGluon-Multimodal (AutoMM): Supercharging Multimodal AutoML with Foundation Modelscited by 30
  24. 2021TransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learningcited by 28
  25. 2023Multi-modal Machine Learning in Engineering Design: A Review and Future Directionscited by 27
  26. 2023LIME: Localized Image Editing via Attention Regularization in Diffusion Modelscited by 27
  27. 2021Dual Progressive Prototype Network for Generalized Zero-Shot Learningcited by 26
  28. 2023AI-Generated Content (AIGC) for Various Data Modalities: A Surveycited by 25
  29. 2024Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detectioncited by 25
  30. 2021Syntactically Guided Generative Embeddings for Zero-Shot Skeleton Action Recognitioncited by 24
  31. 2023Progressive Semantic-Visual Mutual Adaption for Generalized Zero-Shot Learningcited by 24
  32. 2025ICE-Bench: A Unified and Comprehensive Benchmark for Image Creating and Editingcited by 24
  33. 2021Zero-shot and Few-shot Learning with Knowledge Graphs: A Comprehensive Surveycited by 23
  34. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  35. 2022Understanding and Enhancing Robustness of Concept-based Modelscited by 22
  36. 2024ArtFormer: Controllable Generation of Diverse 3D Articulated Objectscited by 22
  37. 2021Large-Scale Zero-Shot Image Classification from Rich and Diverse Textual Descriptionscited by 21
  38. 2023Large Language Models are Good Prompt Learners for Low-Shot Image Classificationcited by 21

50 most cited of 182 All 182 (JSON)