# Canonicalized Stable-List Replay for Private Federated Continual Learning over Language-Model Embeddings

arXiv: https://arxiv.org/abs/2606.00426 · HTML: https://mlsanity.com/paper/2606.00426/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 29 May 2026.

## Builds on (most cited first, 18 of 18)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Deep Learning on a Data Diet: Finding Important Examples Early in Training](https://mlsanity.com/paper/2107.07075/) (2021)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [FedProto: Federated Prototype Learning across Heterogeneous Clients](https://mlsanity.com/paper/2105.00243/) (2021)
- [Federated Continual Learning with Weighted Inter-client Transfer](https://mlsanity.com/paper/2003.03196/) (2020)
- [Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe](https://mlsanity.com/paper/2210.14348/) (2022)
- [Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairness](https://mlsanity.com/paper/2010.01285/) (2020)
- [Better Generative Replay for Continual Federated Learning](https://mlsanity.com/paper/2302.13001/) (2023)
- [A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Tasks](https://mlsanity.com/paper/2311.07784/) (2023)
- [Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning](https://mlsanity.com/paper/2409.01128/) (2024)
- [Accurate Forgetting for Heterogeneous Federated Continual Learning](https://mlsanity.com/paper/2502.14205/) (2025)
- [User-Level Private Learning via Correlated Sampling](https://mlsanity.com/paper/2110.11208/) (2021)
- [Agnostic Private Density Estimation for GMMs via List Global Stability](https://mlsanity.com/paper/2407.04783/) (2024)
- [Using Diffusion Models as Generative Replay in Continual Federated Learning -- What will Happen?](https://mlsanity.com/paper/2411.06618/) (2024)

## Abstract (arXiv)

Federated continual learning (FCL) lets distributed clients adapt language-model heads to evolving NLP tasks without sharing raw text. Under user-level differential privacy (DP), replay-based continual learning faces a structural obstacle: clients can release only small noisy lists of candidate replay summaries, and those lists are unordered across clients. We introduce Canonicalized Stable-List Replay (CSLR), where clients privately produce candidate replay distributions over a shared sentence-embedding space and the server aligns them using signatures induced by public anchor sentences. The anchors provide identifiability for aggregation rather than additional replay data. We prove that, under an observable anchor-signature margin, O(log(N/η)/p) anchors distinguish N candidate list elements with probability at least 1-η, and we give a scoped anchorless non-identifiability result for unordered-label oracle models. Across five seeds on continual classification, NER, and dialogue benchmarks, CSLR improves the final average task metric by 3.9--5.6 points over the strongest non-CSLR DP baseline at eps=4 under the reported replay-release budget, while also outperforming Hungarian and optimal-transport matchers. The formal privacy guarantee covers replay release; end-to-end private training additionally requires composition with a private optimizer for task-head updates.