mlsanityData to 14 Oct 2026

Multimodal Metadata Assignment for Cultural Heritage Artifacts

arXiv 2406.00423 ↗First posted 1 Jun 2024cs.CV cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

We develop a multimodal classifier for the cultural heritage domain using a late fusion approach and introduce a novel dataset. The three modalities are Image, Text, and Tabular data. We based the image classifier on a ResNet convolutional neural network architecture and the text classifier on a multilingual transformer architecture (XML-Roberta). Both are trained as multitask classifiers and use the focal loss to handle class imbalance. Tabular data and late fusion are handled by Gradient Tree Boosting. We also show how we leveraged specific data models and taxonomy in a Knowledge Graph to create the dataset and to store classification results. All individual classifiers accurately predict missing properties in the digitized silk artifacts, with the multimodal approach providing the best results.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 26 of its 69 references that are in the map. First posted on arXiv 1 Jun 2024.

Builds on

26 of 69 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,501
  11. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  12. 2016Enriching Word Vectors with Subword Informationcited by 1,583
14 more
  1. 2014How transferable are features in deep neural networks?cited by 1,225
  2. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 803
  3. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  4. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  5. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  6. 2014CNN Features off-the-shelf: an Astounding Baseline for Recognitioncited by 316
  7. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  8. 2018How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrievalcited by 74
  9. 2018Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterioncited by 70
  10. 2022CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classificationcited by 65
  11. 2021Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distributioncited by 35
  12. 2020Deep Learning on Knowledge Graph for Recommender System: A Surveycited by 19
  13. 2017OmniArt: Multi-task Deep Learning for Artistic Data Analysiscited by 16
  14. 2019The iMet Collection 2019 Challenge Datasetcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Document Classification Pattern Recognition via Information Fusion: A Systematic Review of Multimodal and Multiview Representation Approachescited by 1
  2. 2026M-ArtAgent: Evidence-Based Multimodal Agent for Implicit Art Influence Discoverycited by 0
  3. 2026MinhwaNet: Faithful but Insufficient Object Grounding in Korean Folk Paintingcited by 0
  4. 2026AmalthAI: An Open-Source Computer Vision Platform for Cultural Heritagecited by 0
  5. 2026FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditingcited by 0