# Learning from Aggregated Data: Curated Bags versus Random Bags

arXiv: https://arxiv.org/abs/2305.09557 · HTML: https://mlsanity.com/paper/2305.09557/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 16 May 2023.

## Builds on (most cited first, 9 of 9)

- [A vector-contraction inequality for Rademacher complexities](https://mlsanity.com/paper/1605.00251/) (2016)
- [Deep Learning with Label Differential Privacy](https://mlsanity.com/paper/2102.06062/) (2021)
- [Towards the Science of Security and Privacy in Machine Learning](https://mlsanity.com/paper/1611.03814/) (2016)
- [Privacy Preserving Machine Learning: Threats and Solutions](https://mlsanity.com/paper/1804.11238/) (2018)
- [Private Learning and Sanitization: Pure vs. Approximate Differential Privacy](https://mlsanity.com/paper/1407.2674/) (2014)
- [An Overview of Privacy in Machine Learning](https://mlsanity.com/paper/2005.08679/) (2020)
- [On Learning from Label Proportions](https://mlsanity.com/paper/1402.5902/) (2014)
- [Learning from Aggregate Observations](https://mlsanity.com/paper/2004.06316/) (2020)
- [Lessons from the AdKDD'21 Privacy-Preserving ML Challenge](https://mlsanity.com/paper/2201.13123/) (2022)

## Built on by (most cited first, 11 of 11)

- [PAC Learning Linear Thresholds from Label Proportions](https://mlsanity.com/paper/2310.10098/) (2023)
- [Learning from Label Proportions: Bootstrapping Supervised Learners via Belief Propagation](https://mlsanity.com/paper/2310.08056/) (2023)
- [LLP-Bench: A Large Scale Tabular Benchmark for Learning from Label Proportions](https://mlsanity.com/paper/2310.10096/) (2023)
- [Optimistic Rates for Learning from Label Proportions](https://mlsanity.com/paper/2406.00487/) (2024)
- [Learning from Aggregate responses: Instance Level versus Bag Level Loss Functions](https://mlsanity.com/paper/2401.11081/) (2024)
- [PriorBoost: An Adaptive Algorithm for Learning from Aggregate Responses](https://mlsanity.com/paper/2402.04987/) (2024)
- [Label Differential Privacy via Aggregation](https://mlsanity.com/paper/2310.10092/) (2023)
- [Hardness of Learning Boolean Functions from Label Proportions](https://mlsanity.com/paper/2403.19401/) (2024)
- [Weak to Strong Learning from Aggregate Labels](https://mlsanity.com/paper/2411.06200/) (2024)
- [Learning from Label Proportions and Covariate-shifted Instances](https://mlsanity.com/paper/2411.12334/) (2024)
- [Aggregating Data for Optimal and Private Learning](https://mlsanity.com/paper/2411.19045/) (2024)

## Abstract (arXiv)

Protecting user privacy is a major concern for many machine learning systems that are deployed at scale and collect from a diverse set of population. One way to address this concern is by collecting and releasing data labels in an aggregated manner so that the information about a single user is potentially combined with others. In this paper, we explore the possibility of training machine learning models with aggregated data labels, rather than individual labels. Specifically, we consider two natural aggregation procedures suggested by practitioners: curated bags where the data points are grouped based on common features and random bags where the data points are grouped randomly in bag of similar sizes. For the curated bag setting and for a broad range of loss functions, we show that we can perform gradient-based learning without any degradation in performance that may result from aggregating data. Our method is based on the observation that the sum of the gradients of the loss function on individual data examples in a curated bag can be computed from the aggregate label without the need for individual labels. For the random bag setting, we provide a generalization risk bound based on the Rademacher complexity of the hypothesis class and show how empirical risk minimization can be regularized to achieve the smallest risk bound. In fact, in the random bag setting, there is a trade-off between size of the bag and the achievable error rate as our bound indicates. Finally, we conduct a careful empirical study to confirm our theoretical findings. In particular, our results suggest that aggregate learning can be an effective method for preserving user privacy while maintaining model accuracy.