# Word Embedding Dimension Reduction via Weakly-Supervised Feature Selection

arXiv: https://arxiv.org/abs/2407.12342 · HTML: https://mlsanity.com/paper/2407.12342/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [Simple and Effective Dimensionality Reduction for Word Embeddings](https://mlsanity.com/paper/1708.03629/) (2017)
- [Evaluating Word Embedding Models: Methods and Experimental Results](https://mlsanity.com/paper/1901.09785/) (2019)
- [Compressing Word Embeddings via Deep Compositional Code Learning](https://mlsanity.com/paper/1711.01068/) (2017)
- [Green Learning: Introduction, Examples and Outlook](https://mlsanity.com/paper/2210.00965/) (2022)
- [Problems With Evaluation of Word Embeddings Using Word Similarity Tasks](https://mlsanity.com/paper/1605.02276/) (2016)
- [On Supervised Feature Selection from High Dimensional Feature Spaces](https://mlsanity.com/paper/2203.11924/) (2022)
- [An Overview on Language Models: Recent Developments and Outlook](https://mlsanity.com/paper/2303.05759/) (2023)
- [Bias and Fairness in Chatbots: An Overview](https://mlsanity.com/paper/2309.08836/) (2023)
- [Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings](https://mlsanity.com/paper/2403.14001/) (2024)
- [On Geodesic Distances and Contextual Embedding Compression for Text Classification](https://mlsanity.com/paper/2104.11295/) (2021)

## Abstract (arXiv)

As a fundamental task in natural language processing, word embedding converts each word into a representation in a vector space. A challenge with word embedding is that as the vocabulary grows, the vector space's dimension increases, which can lead to a vast model size. Storing and processing word vectors are resource-demanding, especially for mobile edge-devices applications. This paper explores word embedding dimension reduction. To balance computational costs and performance, we propose an efficient and effective weakly-supervised feature selection method named WordFS. It has two variants, each utilizing novel criteria for feature selection. Experiments on various tasks (e.g., word and sentence similarity and binary and multi-class classification) indicate that the proposed WordFS model outperforms other dimension reduction methods at lower computational costs. We have released the code for reproducibility along with the paper.