mlsanityData to 14 Oct 2026

Semantic Compression for Word and Sentence Embeddings using Discrete Wavelet Transform

arXiv 2508.00220 ↗First posted 31 Jul 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Wavelet transforms, a powerful mathematical tool, have been widely used in different domains, including Signal and Image processing, to unravel intricate patterns, enhance data representation, and extract meaningful features from data. Tangible results from their application suggest that Wavelet transforms can be applied to NLP capturing a variety of linguistic and semantic properties. In this paper, we empirically leverage the application of Discrete Wavelet Transforms (DWT) to word and sentence embeddings. We aim to showcase the capabilities of DWT in analyzing embedding representations at different levels of resolution and compressing them while maintaining their overall quality. We assess the effectiveness of DWT embeddings on semantic similarity tasks to show how DWT can be used to consolidate important semantic information in an embedding vector. We show the efficacy of the proposed paradigm using different embedding models, including large language models, on downstream tasks. Our results show that DWT can reduce the dimensionality of embeddings by 50-93% with almost no change in performance for semantic similarity tasks, while achieving superior accuracy in most downstream tasks. Our findings pave the way for applying DWT to improve NLP applications.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 60 references that are in the map. First posted on arXiv 31 Jul 2025.

Builds on

20 of 60 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  5. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  6. 2015Recent Advances in Convolutional Neural Networkscited by 360
  7. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  8. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  9. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 172
  10. 2014A Tutorial on Principal Component Analysiscited by 141
  11. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  12. 2020SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Modelscited by 56
8 more
  1. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  2. 2017Simple and Effective Dimensionality Reduction for Word Embeddingscited by 39
  3. 2017Compressing Word Embeddings via Deep Compositional Code Learningcited by 35
  4. 2018Online Embedding Compression for Text Classification using Low Rank Matrix Factorizationcited by 22
  5. 2023On the Dimensionality of Sentence Embeddingscited by 5
  6. 2021Direction is what you need: Improving Word Embedding Compression in Large Language Modelscited by 4
  7. 2020Sentence Analogies: Exploring Linguistic Relationships and Regularities in Sentence Embeddingscited by 3
  8. 2021DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddingscited by 2

Built on by

Not cited yet.