# Group-Sparse Matrix Factorization for Transfer Learning of Word Embeddings

arXiv: https://arxiv.org/abs/2104.08928 · HTML: https://mlsanity.com/paper/2104.08928/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 42 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overview](https://mlsanity.com/paper/1809.09573/) (2018)
- [No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis](https://mlsanity.com/paper/1704.00708/) (2017)
- [Global Optimality of Local Search for Low Rank Matrix Recovery](https://mlsanity.com/paper/1605.07221/) (2016)
- [Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima](https://mlsanity.com/paper/1305.2436/) (2013)
- [The Implicit Regularization of Stochastic Gradient Flow for Least Squares](https://mlsanity.com/paper/2003.07802/) (2020)
- [Predicting with Proxies: Transfer Learning in High Dimension](https://mlsanity.com/paper/1812.11097/) (2018)
- [Learning Eligibility in Cancer Clinical Trials using Deep Neural Networks](https://mlsanity.com/paper/1803.08312/) (2018)
- [Mittens: An Extension of GloVe for Learning Domain-Specialized Representations](https://mlsanity.com/paper/1803.09901/) (2018)
- [Domain Adapted Word Embeddings for Improved Sentiment Classification](https://mlsanity.com/paper/1805.04576/) (2018)
- [A Simple Regularization-based Algorithm for Learning Cross-Domain Word Embeddings](https://mlsanity.com/paper/1902.00184/) (2019)

## Built on by (most cited first, 6 of 6)

- [Multitask Learning and Bandits via Robust Statistics](https://mlsanity.com/paper/2112.14233/) (2021)
- [Optimal Multitask Linear Regression and Contextual Bandits under Sparse Heterogeneity](https://mlsanity.com/paper/2306.06291/) (2023)
- [Robust Generalization of Quadratic Neural Networks via Function Identification](https://mlsanity.com/paper/2109.10935/) (2021)
- [LEARNER: A Transfer Learning Method for Low-Rank Matrix Estimation](https://mlsanity.com/paper/2412.20605/) (2024)
- [Stochastic Bandits with ReLU Neural Networks](https://mlsanity.com/paper/2405.07331/) (2024)
- [A Probabilistic Approach for Model Alignment with Human Comparisons](https://mlsanity.com/paper/2403.10771/) (2024)

## Abstract (arXiv)

Unstructured text provides decision-makers with a rich data source in many domains, ranging from product reviews in retail to nursing notes in healthcare. To leverage this information, words are typically translated into word embeddings -- vectors that encode the semantic relationships between words -- through unsupervised learning algorithms such as matrix factorization. However, learning word embeddings from new domains with limited training data can be challenging, because the meaning/usage may be different in the new domain, e.g., the word ``positive'' typically has positive sentiment, but often has negative sentiment in medical notes since it may imply that a patient tested positive for a disease. In practice, we expect that only a small number of domain-specific words may have new meanings. We propose an intuitive two-stage estimator that exploits this structure via a group-sparse penalty to efficiently transfer learn domain-specific word embeddings by combining large-scale text corpora (such as Wikipedia) with limited domain-specific text data. We bound the generalization error of our transfer learning estimator, proving that it can achieve high accuracy with substantially less domain-specific data when only a small number of embeddings are altered between domains. Furthermore, we prove that all local minima identified by our nonconvex objective function are statistically indistinguishable from the global minimum under standard regularization conditions, implying that our estimator can be computed efficiently. Our results provide the first bounds on group-sparse matrix factorization, which may be of independent interest. We empirically evaluate our approach compared to state-of-the-art fine-tuning heuristics from natural language processing.