# RD-DPP: Rate-Distortion Theory Meets Determinantal Point Process to Diversify Learning Data Samples

arXiv: https://arxiv.org/abs/2304.04137 · HTML: https://mlsanity.com/paper/2304.04137/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 54 references that are in the map.
- First posted on arXiv 9 Apr 2023.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Prevalence of Neural Collapse during the terminal phase of deep learning training](https://mlsanity.com/paper/2008.08186/) (2020)
- [Selection via Proxy: Efficient Data Selection for Deep Learning](https://mlsanity.com/paper/1906.11829/) (2019)
- [Coresets via Bilevel Optimization for Continual Learning and Streaming](https://mlsanity.com/paper/2006.03875/) (2020)
- [Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction](https://mlsanity.com/paper/2006.08558/) (2020)
- [Determinantal Point Processes in Randomized Numerical Linear Algebra](https://mlsanity.com/paper/2005.03185/) (2020)
- [Continuum: Simple Management of Complex Continual Learning Scenarios](https://mlsanity.com/paper/2102.06253/) (2021)
- [In Defense of Core-set: A Density-aware Core-set Selection for Active Learning](https://mlsanity.com/paper/2206.04838/) (2022)
- [Exact sampling of determinantal point processes with sublinear time preprocessing](https://mlsanity.com/paper/1905.13476/) (2019)
- [Determinantal Point Processes for Coresets](https://mlsanity.com/paper/1803.08700/) (2018)
- [Sampling from a k-DPP without looking at all items](https://mlsanity.com/paper/2006.16947/) (2020)
- [Fast determinantal point processes via distortion-free intermediate sampling](https://mlsanity.com/paper/1811.03717/) (2018)
- [Wireless Data Acquisition for Edge Learning: Data-Importance Aware Retransmission](https://mlsanity.com/paper/1812.02030/) (2018)
- [Submodular Mutual Information for Targeted Data Subset Selection](https://mlsanity.com/paper/2105.00043/) (2021)
- [Minimum-Margin Active Learning](https://mlsanity.com/paper/1906.00025/) (2019)
- [Lazy and Fast Greedy MAP Inference for Determinantal Point Process](https://mlsanity.com/paper/2206.05947/) (2022)

## Built on by (most cited first, 4 of 4)

- [DGR-MIL: Exploring Diverse Global Representation in Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2407.03575/) (2024)
- [MTS-LOF: Medical Time-Series Representation Learning via Occlusion-Invariant Features](https://mlsanity.com/paper/2310.12451/) (2023)
- [Learning on Bandwidth Constrained Multi-Source Data with MIMO-inspired DPP MAP Inference](https://mlsanity.com/paper/2306.02497/) (2023)
- [Reliability-Aware Determinantal Point Processes for Robust Informative Data Selection in Large Language Models](https://mlsanity.com/paper/2602.00885/) (2026)

## Abstract (arXiv)

In some practical learning tasks, such as traffic video analysis, the number of available training samples is restricted by different factors, such as limited communication bandwidth and computation power. Determinantal Point Process (DPP) is a common method for selecting the most diverse samples to enhance learning quality. However, the number of selected samples is restricted to the rank of the kernel matrix implied by the dimensionality of data samples. Secondly, it is not easily customizable to different learning tasks. In this paper, we propose a new way of measuring task-oriented diversity based on the Rate-Distortion (RD) theory, appropriate for multi-level classification. To this end, we establish a fundamental relationship between DPP and RD theory. We observe that the upper bound of the diversity of data selected by DPP has a universal trend of textitphase transition, which suggests that DPP is beneficial only at the beginning of sample accumulation. This led to the design of a bi-modal method, where RD-DPP is used in the first mode to select initial data samples, then classification inconsistency (as an uncertainty measure) is used to select the subsequent samples in the second mode. This phase transition solves the limitation to the rank of the similarity matrix. Applying our method to six different datasets and five benchmark models suggests that our method consistently outperforms random selection, DPP-based methods, and alternatives like uncertainty-based and coreset methods under all sampling budgets, while exhibiting high generalizability to different learning tasks.