# A Bag-of-Prototypes Representation for Dataset-Level Applications arXiv: https://arxiv.org/abs/2303.13251 ยท HTML: https://mlsanity.com/paper/2303.13251/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 48 of its 100 references that are in the map. - First posted on arXiv 23 Mar 2023. ## Builds on (most cited first, 30 of 48) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016) - [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019) - [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018) - [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019) - [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016) - [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018) - [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018) - [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016) - [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019) - [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018) - [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018) - [Data Shapley: Equitable Valuation of Data for Machine Learning](https://mlsanity.com/paper/1904.02868/) (2019) - [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014) - [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021) - [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020) - [Return of Frustratingly Easy Domain Adaptation](https://mlsanity.com/paper/1511.05547/) (2015) - [Bridging Theory and Algorithm for Domain Adaptation](https://mlsanity.com/paper/1904.05801/) (2019) - [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020) - [Meta-Learning: A Survey](https://mlsanity.com/paper/1810.03548/) (2018) - [Task2Vec: Task Embedding for Meta-Learning](https://mlsanity.com/paper/1902.03545/) (2019) - [Do CIFAR-10 Classifiers Generalize to CIFAR-10?](https://mlsanity.com/paper/1806.00451/) (2018) - [Geometric Dataset Distances via Optimal Transport](https://mlsanity.com/paper/2002.02923/) (2020) - [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022) - [StyleGAN-Human: A Data-Centric Odyssey of Human Generation](https://mlsanity.com/paper/2204.11823/) (2022) - [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021) - [Predicting with Confidence on Unseen Distributions](https://mlsanity.com/paper/2107.03315/) (2021) ## Built on by (most cited first, 11 of 11) - [CIFAR-10-Warehouse: Broad and More Realistic Testbeds in Model Generalization Analysis](https://mlsanity.com/paper/2310.04414/) (2023) - [MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts](https://mlsanity.com/paper/2405.18979/) (2024) - [What Does Softmax Probability Tell Us about Classifiers Ranking Across Diverse Test Conditions?](https://mlsanity.com/paper/2406.09908/) (2024) - [Ranked from Within: Ranking Large Multimodal Models Without Labels](https://mlsanity.com/paper/2412.06461/) (2024) - [Suitability Filter: A Statistical Framework for Classifier Evaluation in Real-World Deployment Settings](https://mlsanity.com/paper/2505.22356/) (2025) - [Provably Improving Generalization of Few-Shot Models with Synthetic Data](https://mlsanity.com/paper/2505.24190/) (2025) - [What you see is what you get: Experience ranking with deep neural dataset-to-dataset similarity for topological localisation](https://mlsanity.com/paper/2310.13622/) (2023) - [Poor-Supervised Evaluation for SuperLLM via Mutual Consistency](https://mlsanity.com/paper/2408.13738/) (2024) - [Bipartite Mode Matching for Vision Training Set Search from a Hierarchical Data Server](https://mlsanity.com/paper/2601.09531/) (2026) - [GFPL: Generative Federated Prototype Learning for Resource-Constrained and Data-Imbalanced Vision Task](https://mlsanity.com/paper/2602.21873/) (2026) - [VDNA-PR: Using General Dataset Representations for Robust Sequential Visual Place Recognition](https://mlsanity.com/paper/2403.09025/) (2024) ## Abstract (arXiv) This work investigates dataset vectorization for two dataset-level tasks: assessing training set suitability and test set difficulty. The former measures how suitable a training set is for a target domain, while the latter studies how challenging a test set is for a learned model. Central to the two tasks is measuring the underlying relationship between datasets. This needs a desirable dataset vectorization scheme, which should preserve as much discriminative dataset information as possible so that the distance between the resulting dataset vectors can reflect dataset-to-dataset similarity. To this end, we propose a bag-of-prototypes (BoP) dataset representation that extends the image-level bag consisting of patch descriptors to dataset-level bag consisting of semantic prototypes. Specifically, we develop a codebook consisting of K prototypes clustered from a reference dataset. Given a dataset to be encoded, we quantize each of its image features to a certain prototype in the codebook and obtain a K-dimensional histogram. Without assuming access to dataset labels, the BoP representation provides a rich characterization of the dataset semantic distribution. Furthermore, BoP representations cooperate well with Jensen-Shannon divergence for measuring dataset-to-dataset similarity. Although very simple, BoP consistently shows its advantage over existing representations on a series of benchmarks for two dataset-level tasks.