mlsanityData to 14 Oct 2026

Canonicalized Stable-List Replay for Private Federated Continual Learning over Language-Model Embeddings

arXiv 2606.00426 ↗First posted 29 May 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated continual learning (FCL) lets distributed clients adapt language-model heads to evolving NLP tasks without sharing raw text. Under user-level differential privacy (DP), replay-based continual learning faces a structural obstacle: clients can release only small noisy lists of candidate replay summaries, and those lists are unordered across clients. We introduce Canonicalized Stable-List Replay (CSLR), where clients privately produce candidate replay distributions over a shared sentence-embedding space and the server aligns them using signatures induced by public anchor sentences. The anchors provide identifiability for aggregation rather than additional replay data. We prove that, under an observable anchor-signature margin, O(log(N/η)/p) anchors distinguish N candidate list elements with probability at least 1-η, and we give a scoped anchorless non-identifiability result for unordered-label oracle models. Across five seeds on continual classification, NER, and dialogue benchmarks, CSLR improves the final average task metric by 3.9--5.6 points over the strongest non-CSLR DP baseline at eps=4 under the reported replay-release budget, while also outperforming Hungarian and optimal-transport matchers. The formal privacy guarantee covers replay release; end-to-end private training additionally requires composition with a private optimizer for task-head updates.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 30 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

18 of 30 references.

References · 18

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2016Deep Learning with Differential Privacycited by 2,728
  3. 2018Experience Replay for Continual Learningcited by 1,095
  4. 2021Model-Contrastive Federated Learningcited by 766
  5. 2017Renyi Differential Privacycited by 650
  6. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  7. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  8. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  9. 2020Federated Continual Learning with Weighted Inter-client Transfercited by 161
  10. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  11. 2020Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairnesscited by 67
  12. 2023Better Generative Replay for Continual Federated Learningcited by 55
6 more
  1. 2023A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Taskscited by 41
  2. 2024Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learningcited by 23
  3. 2025Accurate Forgetting for Heterogeneous Federated Continual Learningcited by 11
  4. 2021User-Level Private Learning via Correlated Samplingcited by 9
  5. 2024Agnostic Private Density Estimation for GMMs via List Global Stabilitycited by 2
  6. 2024Using Diffusion Models as Generative Replay in Continual Federated Learning -- What will Happen?cited by 2

Built on by

Not cited yet.