mlsanityData to 7 Oct 2026

Kaleido-BERT: Vision-Language Pre-training on Fashion Domain

arXiv 2103.16110 ↗First posted 30 Mar 2021cs.CVcited by 67

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

We present a new vision-language (VL) pre-training model dubbed Kaleido-BERT, which introduces a novel kaleido strategy for fashion cross-modality representations from transformers. In contrast to random masking strategy of recent VL models, we design alignment guided masking to jointly focus more on image-text semantic relations. To this end, we carry out five novel tasks, i.e., rotation, jigsaw, camouflage, grey-to-color, and blank-to-color for self-supervised VL pre-training at patches of different scale. Kaleido-BERT is conceptually simple and easy to extend to the existing BERT framework, it attains new state-of-the-art results by large margins on four downstream tasks, including text retrieval (R@1: 4.03% absolute improvement), image retrieval (R@1: 7.13% abs imv.), category recognition (ACC: 3.28% abs imv.), and fashion captioning (Bleu4: 1.2 abs imv.). We validate the efficiency of Kaleido-BERT on a wide range of e-commerical websites, demonstrating its broader potential in real-world applications.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 74 of its 84 references that are in the map. First posted on arXiv 30 Mar 2021.

Builds on

74 of 84 references.

References · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
38 more
  1. 2017The Kinetics Human Action Video Datasetcited by 1,894
  2. 2020Rethinking Attention with Performerscited by 1,584
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  6. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  13. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  14. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  15. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
  16. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  17. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  18. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  19. 2017Structure-measure: A New Way to Evaluate Foreground Mapscited by 480
  20. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  21. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  22. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  23. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  24. 2016Visual Dialogcited by 377
  25. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  26. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  27. 2016Visual Relationship Detection with Language Priorscited by 333
  28. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  29. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  30. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  31. 2015Stacked Attention Networks for Image Question Answeringcited by 273
  32. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  33. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  34. 2019EGNet:Edge Guidance Network for Salient Object Detectioncited by 237
  35. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  36. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  37. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  38. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 21; 2023: 19; 2024: 12; 2025: 5; 2026: 3.

67 papers.

Sort:

By area:

  1. 2022Multimodal Learning with Transformers: A Surveycited by 399
  2. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  3. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  4. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  5. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokenscited by 116
  6. 2021Full-Duplex Strategy for Video Object Segmentationcited by 78
  7. 2022Image-text Retrieval: A Survey on Recent Research and Developmentcited by 55
  8. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  9. 2022Vision-and-Language Pretrained Models: A Surveycited by 50
  10. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  11. 2023Transformers in Speech Processing: A Surveycited by 38
  12. 2024Goldfish: Vision-Language Understanding of Arbitrarily Long Videoscited by 38
38 more
  1. 2023FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Taskscited by 37
  2. 2023Referring Image Segmentation Using Text Supervisioncited by 35
  3. 2022Contrastive Language-Image Pre-Training with Knowledge Graphscited by 34
  4. 2021TransFusion: Cross-view Fusion with Transformer for 3D Human Pose Estimationcited by 32
  5. 2022FashionViL: Fashion-Focused Vision-and-Language Representation Learningcited by 32
  6. 2023S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captionscited by 29
  7. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  8. 2021M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretrainingcited by 26
  9. 2022Progressive Learning for Image Retrieval with Hybrid-Modality Queriescited by 26
  10. 2022CommerceMM: Large-Scale Commerce MultiModal Representation Learning with Omni Retrievalcited by 25
  11. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  12. 2021Recursive Contour Saliency Blending Network for Accurate Salient Object Detectioncited by 20
  13. 2022Skating-Mixer: Long-Term Sport Audio-Visual Modeling with MLPscited by 20
  14. 2021Salient Objects in Cluttercited by 19
  15. 2024UniFashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generationcited by 17
  16. 2021CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classificationcited by 16
  17. 2022Single-Stream Multi-Level Alignment for Vision-Language Pretrainingcited by 16
  18. 2022Masked Vision-Language Transformer in Fashioncited by 14
  19. 2023FashionSAP: Symbols and Attributes Prompt for Fine-grained Fashion Vision-Language Pre-trainingcited by 14
  20. 2022ManiTrans: Entity-Level Text-Guided Image Manipulation via Token-wise Semantic Alignment and Generationcited by 13
  21. 2024Multimodal Learned Sparse Retrieval with Probabilistic Expansion Controlcited by 12
  22. 2024Enhancing Vision-Language Pre-training with Rich Supervisionscited by 12
  23. 2021MLIM: Vision-and-Language Model Pre-training with Masked Language and Image Modelingcited by 11
  24. 2022CapOnImage: Context-driven Dense-Captioning on Imagecited by 11
  25. 2022Video Dialog as Conversation about Objects Living in Space-Timecited by 11
  26. 2023MAKE: Vision-Language Pre-training based Product Retrieval in Taobao Searchcited by 9
  27. 2023OpenFashionCLIP: Vision-and-Language Contrastive Learning with Open-Source Fashion Datacited by 9
  28. 20223D-TOGO: Towards Text-Guided Cross-Category 3D Object Generationcited by 8
  29. 2023Learning Instance-Level Representation for Large-Scale Multi-Modal Pretraining in E-commercecited by 8
  30. 2024SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretrainingcited by 8
  31. 2022FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioningcited by 7
  32. 2025WaMo: Wavelet-Enhanced Multi-Frequency Trajectory Analysis for Fine-Grained Text-Motion Retrievalcited by 7
  33. 2022UIGR: Unified Interactive Garment Retrievalcited by 6
  34. 2022Training and challenging models for text-guided fashion image retrievalcited by 5
  35. 2023Unified Vision-Language Representation Modeling for E-Commerce Same-Style Products Retrievalcited by 5
  36. 2022Understanding Attention for Vision-and-Language Taskscited by 4
  37. 2025ChatIoT: Large Language Model-based Security Assistant for Internet of Things with Retrieval-Augmented Generationcited by 4
  38. 2025Seeing the Abstract: Translating the Abstract Language for Vision Language Modelscited by 4

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022A Survey of Vision-Language Pre-Trained Models
  2. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  3. 2022VL-BEiT: Generative Vision-Language Pretraining
  4. 2023Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment
  5. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
  6. 2023Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
  7. 2023Retrieval-based Knowledge Augmented Vision Language Pre-training
  8. 2021Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation