mlsanityData to 14 Oct 2026

The Curious Layperson: Fine-Grained Image Recognition without Expert Labels

arXiv 2111.03651 ↗First posted 5 Nov 2021cs.CV cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Most of us are not experts in specific fields, such as ornithology. Nonetheless, we do have general image and language understanding capabilities that we use to match what we see to expert resources. This allows us to expand our knowledge and perform novel tasks without ad-hoc external supervision. On the contrary, machines have a much harder time consulting expert-curated knowledge bases unless trained specifically with that knowledge in mind. Thus, in this paper we consider a new problem: fine-grained image recognition without expert annotations, which we address by leveraging the vast knowledge available in web encyclopedias. First, we learn a model to describe the visual appearance of objects using non-expert image descriptions. We then train a fine-grained textual similarity model that matches image descriptions with documents on a sentence-level basis. We evaluate the method on two datasets and compare with several strong baselines and the state of the art in cross-modal retrieval. Code is available at: https://github.com/subhc/clever

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 56 of its 96 references that are in the map. First posted on arXiv 5 Nov 2021.

Builds on

56 of 96 references.

References · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
38 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  3. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  5. 2019Passage Re-ranking with BERTcited by 645
  6. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 538
  7. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  8. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  9. 2020SCAN: Learning to Classify Images without Labelscited by 352
  10. 2019Attention on Attention for Image Captioningcited by 289
  11. 2017Feature Generating Networks for Zero-Shot Learningcited by 257
  12. 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
  13. 2016Learning Deep Representations of Fine-grained Visual Descriptionscited by 182
  14. 2015Label-Embedding for Image Classificationcited by 165
  15. 2019f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learningcited by 164
  16. 2017Semantic Autoencoder for Zero-Shot Learningcited by 133
  17. 2016Latent Embeddings for Zero-shot Classificationcited by 129
  18. 2014Part-based R-CNNs for Fine-grained Category Detectioncited by 109
  19. 2017A Generative Adversarial Approach for Zero-Shot Learning from Noisy Textscited by 96
  20. 2018Rethinking Knowledge Graph Propagation for Zero-Shot Learningcited by 87
  21. 2018Multi-modal Cycle-consistent Generalized Zero-Shot Learningcited by 87
  22. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  23. 2019CEDR: Contextualized Embeddings for Document Rankingcited by 77
  24. 2020Interpretable and Accurate Fine-grained Recognition via Region Groupingcited by 75
  25. 2019Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification from the Bottom Upcited by 66
  26. 2020Leveraging Seen and Unseen Semantic Relationships for Generative Zero-Shot Learningcited by 64
  27. 2019Semantic-Guided Multi-Attention Localization for Zero-Shot Learningcited by 58
  28. 2017Fine-graind Image Classification via Combining Vision and Languagecited by 51
  29. 2015The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognitioncited by 48
  30. 2017Link the head to the "beak": Zero Shot Learning from Noisy Text Description at Part Precisioncited by 46
  31. 2016Less is more: zero-shot learning from online textual documents with noise suppressioncited by 40
  32. 2021A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classificationcited by 36
  33. 2017The iNaturalist Species Classification and Detection Datasetcited by 29
  34. 2017Fine-grained Recognition in the Wild: A Multi-Task Domain Adaptation Approachcited by 27
  35. 2020Compare and Reweight: Distinctive Image Captioning Using Similar Images Setscited by 27
  36. 2015Fine-grained Categorization and Dataset Bootstrapping using Deep Metric Learning with Humans in the Loopcited by 25
  37. 2015Neural Activation Constellations: Unsupervised Part Model Discovery with Convolutional Networkscited by 22
  38. 2017An Overview of Cross-media Retrieval: Concepts, Methodologies, Benchmarks and Challengescited by 21

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 0; 2023: 2; 2024: 1; 2025: 2.

7 papers.

  1. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 101
  2. 2023Multi-Modal Classifiers for Open-Vocabulary Object Detectioncited by 53
  3. 2025Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Modelscited by 31
  4. 2024Democratizing Fine-grained Visual Recognition with Large Language Modelscited by 25
  5. 2021Making a Bird AI Expert Work for You and Mecited by 6
  6. 2025Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Modelcited by 1
  7. 2021Clue Me In: Semi-Supervised FGVC with Out-of-Distribution Datacited by 0