# SMEC: Rethinking Matryoshka Representation Learning for Retrieval Embedding Compression

arXiv: https://arxiv.org/abs/2510.12474 · HTML: https://mlsanity.com/paper/2510.12474/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [NISP: Pruning Networks using Neuron Importance Score Propagation](https://mlsanity.com/paper/1711.05908/) (2017)
- [Unsupervised Learning of Dense Visual Representations](https://mlsanity.com/paper/2011.05499/) (2020)
- [Automatic Spatially-aware Fashion Concept Discovery](https://mlsanity.com/paper/1708.01311/) (2017)
- [Cross-Batch Memory for Embedding Learning](https://mlsanity.com/paper/1912.06798/) (2019)
- [LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation](https://mlsanity.com/paper/2306.11222/) (2023)
- [Products-10K: A Large-scale Product Recognition Dataset](https://mlsanity.com/paper/2008.10545/) (2020)
- [Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions](https://mlsanity.com/paper/2407.20243/) (2024)
- [Search-Adaptor: Embedding Customization for Information Retrieval](https://mlsanity.com/paper/2310.08750/) (2023)

## Built on by (most cited first, 14 of 14)

- [Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval](https://mlsanity.com/paper/2602.19961/) (2026)
- [Efficient Temporal-aware Matryoshka Adaptation for Temporal Information Retrieval](https://mlsanity.com/paper/2601.05549/) (2026)
- [MM-Matryoshka: Towards Budget-Elastic Visual Document Retrieval via a 2D Multimodal Matryoshka Training Framework](https://mlsanity.com/paper/2606.07654/) (2026)
- [3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning](https://mlsanity.com/paper/2608.29285/) (2026)
- [Matryoshka Hash Representations for Model-Aware Compact Semantic Retrieval](https://mlsanity.com/paper/2609.07276/) (2026)
- [Learning to Select: Query-Aware Adaptive Dimension Selection for Dense Retrieval](https://mlsanity.com/paper/2602.03306/) (2026)
- [Cross-lingual Matryoshka Representation Learning across Speech and Text](https://mlsanity.com/paper/2602.19991/) (2026)
- [MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation](https://mlsanity.com/paper/2602.21379/) (2026)
- [ML-Embed: Inclusive and Efficient Embeddings for a Multilingual World](https://mlsanity.com/paper/2605.15081/) (2026)
- [To MRL or not to MRL: Text Embeddings are Robust to Truncation Without Matryoshka Learning, Except In Heavy Truncation Scenarios](https://mlsanity.com/paper/2605.16608/) (2026)
- [GraSP-VL: Length as a Semantic Granularity Interface for Vision-Language Representations](https://mlsanity.com/paper/2605.17727/) (2026)
- [DIVE: Embedding Compression via Self-Limiting Gradient Updates](https://mlsanity.com/paper/2605.20689/) (2026)
- [CAMMAR: Culture-Aware Matryoshka for Metaphorical Arabic Representations](https://mlsanity.com/paper/2607.15847/) (2026)
- [MIND the Gap: A Geographic Implicit Neural Representation with Adjustable Spatial Scale](https://mlsanity.com/paper/2609.25454/) (2026)

## Abstract (arXiv)

Large language models (LLMs) generate high-dimensional embeddings that capture rich semantic and syntactic information. However, high-dimensional embeddings exacerbate computational complexity and storage requirements, thereby hindering practical deployment. To address these challenges, we propose a novel training framework named Sequential Matryoshka Embedding Compression (SMEC). This framework introduces the Sequential Matryoshka Representation Learning(SMRL) method to mitigate gradient variance during training, the Adaptive Dimension Selection (ADS) module to reduce information degradation during dimension pruning, and the Selectable Cross-batch Memory (S-XBM) module to enhance unsupervised learning between high- and low-dimensional embeddings. Experiments on image, text, and multimodal datasets demonstrate that SMEC achieves significant dimensionality reduction while maintaining performance. For instance, on the BEIR dataset, our approach improves the performance of compressed LLM2Vec embeddings (256 dimensions) by 1.1 points and 2.7 points compared to the Matryoshka-Adaptor and Search-Adaptor models, respectively.