# Efficient Sentence Embedding using Discrete Cosine Transform

arXiv: https://arxiv.org/abs/1909.03104 · HTML: https://mlsanity.com/paper/1909.03104/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Sep 2019.

## Built on by (most cited first, 7 of 7)

- [Just Rank: Rethinking Evaluation with Word and Sentence Similarities](https://mlsanity.com/paper/2203.02679/) (2022)
- [A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyond](https://mlsanity.com/paper/2305.12641/) (2023)
- [Unsupervised Sentence-embeddings by Manifold Approximation and Projection](https://mlsanity.com/paper/2102.03795/) (2021)
- [Discrete Cosine Transform as Universal Sentence Encoder](https://mlsanity.com/paper/2106.00934/) (2021)
- [Clustering and Network Analysis for the Embedding Spaces of Sentences and Sub-Sentences](https://mlsanity.com/paper/2110.00697/) (2021)
- [NoPPA: Non-Parametric Pairwise Attention Random Walk Model for Sentence Representation](https://mlsanity.com/paper/2302.12903/) (2023)
- [Combining Discrete Wavelet and Cosine Transforms for Efficient Sentence Embedding](https://mlsanity.com/paper/2508.00420/) (2025)

## Abstract (arXiv)

Vector averaging remains one of the most popular sentence embedding methods in spite of its obvious disregard for syntactic structure. While more complex sequential or convolutional networks potentially yield superior classification performance, the improvements in classification accuracy are typically mediocre compared to the simple vector averaging. As an efficient alternative, we propose the use of discrete cosine transform (DCT) to compress word sequences in an order-preserving manner. The lower order DCT coefficients represent the overall feature patterns in sentences, which results in suitable embeddings for tasks that could benefit from syntactic features. Our results in semantic probing tasks demonstrate that DCT embeddings indeed preserve more syntactic information compared with vector averaging. With practically equivalent complexity, the model yields better overall performance in downstream classification tasks that correlate with syntactic features, which illustrates the capacity of DCT to preserve word order information.