# ArtiFact: A Large-Scale Multi-Modal Cultural Heritage Dataset

arXiv: https://arxiv.org/abs/2606.09648 · HTML: https://mlsanity.com/paper/2606.09648/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 48 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspective](https://mlsanity.com/paper/2112.06409/) (2021)
- [HoloClean: Holistic Data Repairs with Probabilistic Inference](https://mlsanity.com/paper/1702.00820/) (2017)
- [How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval](https://mlsanity.com/paper/1810.09617/) (2018)
- [All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages](https://mlsanity.com/paper/2411.16508/) (2024)
- [Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models](https://mlsanity.com/paper/2502.16033/) (2025)
- [From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalities](https://mlsanity.com/paper/2412.11694/) (2024)
- [OmniArt: Multi-task Deep Learning for Artistic Data Analysis](https://mlsanity.com/paper/1708.00684/) (2017)
- [DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culture](https://mlsanity.com/paper/2509.19274/) (2025)
- [Time Travel: A Comprehensive Benchmark to Evaluate LMMs on Historical and Cultural Artifacts](https://mlsanity.com/paper/2502.14865/) (2025)
- [DataVinci: Learning Syntactic and Semantic String Repairs](https://mlsanity.com/paper/2308.10922/) (2023)
- [LEMoN: Label Error Detection using Multimodal Neighbors](https://mlsanity.com/paper/2407.18941/) (2024)
- [Towards Cross-Modal Error Detection with Tables and Images](https://mlsanity.com/paper/2510.12383/) (2025)

## Abstract (arXiv)

Multi-modal data management has emerged as a central research topic in the database community, spanning data integration, semantic query processing, and data quality assessment. Despite this growing interest, the community lacks large-scale, real-world datasets combining tables, text, and images. We present ArtiFact, a multi-modal cultural heritage dataset of 651045 museum records collected from the Metropolitan Museum of Art, the Art Institute of Chicago, and the Rijksmuseum. We demonstrate the utility of ArtiFact through two downstream tasks. For cross-modal error detection, we introduce a curated taxonomy of seven error categories injected into 130209 records and show that reliably detecting subtle domain-specific errors such as material anachronisms and temporal shifts remain an open challenge. For semantic query processing, we show that current systems struggle with queries involving cultural proximity, ambiguous object types, and historically contingent terminology. Our results position ArtiFact as a challenging benchmark for multi-modal data management research.