# Learning Federated Representations and Recommendations with Limited Negatives

arXiv: https://arxiv.org/abs/2108.07931 · HTML: https://mlsanity.com/paper/2108.07931/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 18 Aug 2021.

## Builds on (most cited first, 16 of 16)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [Efficient Natural Language Response Suggestion for Smart Reply](https://mlsanity.com/paper/1705.00652/) (2017)
- [Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning](https://mlsanity.com/paper/2008.03606/) (2020)
- [Learning Dense Representations for Entity Retrieval](https://mlsanity.com/paper/1909.10506/) (2019)
- [FedMix: Approximation of Mixup under Mean Augmented Federated Learning](https://mlsanity.com/paper/2107.00233/) (2021)
- [Federated Learning with Only Positive Labels](https://mlsanity.com/paper/2004.10342/) (2020)
- [Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model](https://mlsanity.com/paper/1810.12836/) (2018)
- [End-to-End Retrieval in Continuous Space](https://mlsanity.com/paper/1811.08008/) (2018)
- [Learning Semantic Textual Similarity from Conversations](https://mlsanity.com/paper/1804.07754/) (2018)
- [Learning Spread-out Local Feature Descriptors](https://mlsanity.com/paper/1708.06320/) (2017)
- [Length bias in Encoder Decoder Models and a Case for Global Conditioning](https://mlsanity.com/paper/1606.03402/) (2016)

## Built on by (most cited first, 6 of 6)

- [User-LLM: Efficient LLM Contextualization with User Embeddings](https://mlsanity.com/paper/2402.13598/) (2024)
- [Mixed Federated Learning: Joint Decentralized and Centralized Learning](https://mlsanity.com/paper/2205.13655/) (2022)
- [FedCL: Federated Contrastive Learning for Privacy-Preserving Recommendation](https://mlsanity.com/paper/2204.09850/) (2022)
- [Robust Convergence in Federated Learning through Label-wise Clustering](https://mlsanity.com/paper/2112.14244/) (2021)
- [Federated Training of Dual Encoding Models on Small Non-IID Client Datasets](https://mlsanity.com/paper/2210.00092/) (2022)
- [Towards Federated Learning Under Resource Constraints via Layer-wise Training and Depth Dropout](https://mlsanity.com/paper/2309.05213/) (2023)

## Abstract (arXiv)

Deep retrieval models are widely used for learning entity representations and recommendations. Federated learning provides a privacy-preserving way to train these models without requiring centralization of user data. However, federated deep retrieval models usually perform much worse than their centralized counterparts due to non-IID (independent and identically distributed) training data on clients, an intrinsic property of federated learning that limits negatives available for training. We demonstrate that this issue is distinct from the commonly studied client drift problem. This work proposes batch-insensitive losses as a way to alleviate the non-IID negatives issue for federated movie recommendations. We explore a variety of techniques and identify that batch-insensitive losses can effectively improve the performance of federated deep retrieval models, increasing the relative recall of the federated model by up to 93.15% and reducing the relative gap in recall between it and a centralized model from 27.22% - 43.14% to 0.53% - 2.42%. We also open-source our code framework to accelerate further research and applications of federated deep retrieval models.