mlsanityData to 7 Oct 2026

Now It Sounds Like You: Learning Personalized Vocabulary On Device

arXiv 2305.03584 ↗First posted 5 May 2023cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

In recent years, Federated Learning (FL) has shown significant advancements in its ability to perform various natural language processing (NLP) tasks. This work focuses on applying personalized FL for on-device language modeling. Due to limitations of memory and latency, these models cannot support the complexity of sub-word tokenization or beam search decoding, resulting in the decision to deploy a closed-vocabulary language model. However, closed-vocabulary models are unable to handle out-of-vocabulary (OOV) words belonging to specific users. To address this issue, We propose a novel technique called "OOV expansion" that improves OOV coverage and increases model accuracy while minimizing the impact on memory and latency. This method introduces a personalized "OOV adapter" that effectively transfers knowledge from a central model and learns word embedding for personalized vocabulary. OOV expansion significantly outperforms standard FL personalization methods on a set of common FL benchmarks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 36 of its 46 references that are in the map. First posted on arXiv 5 May 2023.

Builds on

36 of 46 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  9. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  11. 2018LEAF: A Benchmark for Federated Settingscited by 780
  12. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
24 more
  1. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  2. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  3. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  4. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  5. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  6. 2020Federated Learning of a Mixture of Global and Local Modelscited by 208
  7. 2019Federated Evaluation of On-device Personalizationcited by 147
  8. 2020Survey of Personalization Techniques for Federated Learningcited by 144
  9. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  10. 2015Character-Aware Neural Language Modelscited by 129
  11. 2022Federated Learning with Partial Model Personalizationcited by 114
  12. 2019Gmail Smart Compose: Real-Time Assisted Writingcited by 97
  13. 2021FedNLP: Benchmarking Federated Learning Methods for Natural Language Processing Taskscited by 84
  14. 2021Federated Reconstruction: Partially Local Federated Learningcited by 78
  15. 2019Federated Learning Of Out-Of-Vocabulary Wordscited by 51
  16. 2022ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsitycited by 45
  17. 2021Federated Learning Meets Natural Language Processing: A Surveycited by 42
  18. 2021Scaling Federated Learning for Fine-tuning of Large Language Modelscited by 36
  19. 2019Federated Learning of N-gram Language Modelscited by 33
  20. 2022Scaling Language Model Size in Cross-Device Federated Learningcited by 25
  21. 2021On-device Federated Learning with Flowercited by 13
  22. 2021Remember the context! ASR slot error correction through memorizationcited by 9
  23. 2023Green Federated Learningcited by 8
  24. 2022Training a Tokenizer for Free with Private Federated Learningcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2023Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyondcited by 22
  2. 2024On the Linear Speedup of Personalized Federated Reinforcement Learning with Shared Representationscited by 4