# Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions

arXiv: https://arxiv.org/abs/2407.20243 · HTML: https://mlsanity.com/paper/2407.20243/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 17 Jul 2024.

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://mlsanity.com/paper/2212.09741/) (2022)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://mlsanity.com/paper/2209.11755/) (2022)
- [Generative Representational Instruction Tuning](https://mlsanity.com/paper/2402.09906/) (2024)
- [Automatic Spatially-aware Fashion Concept Discovery](https://mlsanity.com/paper/1708.01311/) (2017)
- [Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages](https://mlsanity.com/paper/2210.09984/) (2022)

## Built on by (most cited first, 16 of 16)

- [DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruning](https://mlsanity.com/paper/2509.23883/) (2025)
- [SMEC: Rethinking Matryoshka Representation Learning for Retrieval Embedding Compression](https://mlsanity.com/paper/2510.12474/) (2025)
- [Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval](https://mlsanity.com/paper/2602.19961/) (2026)
- [CSRv2: Unlocking Ultra-Sparse Embeddings](https://mlsanity.com/paper/2602.05735/) (2026)
- [Efficient Temporal-aware Matryoshka Adaptation for Temporal Information Retrieval](https://mlsanity.com/paper/2601.05549/) (2026)
- [MM-Matryoshka: Towards Budget-Elastic Visual Document Retrieval via a 2D Multimodal Matryoshka Training Framework](https://mlsanity.com/paper/2606.07654/) (2026)
- [Matryoshka Hash Representations for Model-Aware Compact Semantic Retrieval](https://mlsanity.com/paper/2609.07276/) (2026)
- [ConceptCarve: Dynamic Realization of Evidence](https://mlsanity.com/paper/2504.07228/) (2025)
- [Compressed Concatenation of Small Embedding Models](https://mlsanity.com/paper/2510.04626/) (2025)
- [Learning to Select: Query-Aware Adaptive Dimension Selection for Dense Retrieval](https://mlsanity.com/paper/2602.03306/) (2026)
- [ML-Embed: Inclusive and Efficient Embeddings for a Multilingual World](https://mlsanity.com/paper/2605.15081/) (2026)
- [To MRL or not to MRL: Text Embeddings are Robust to Truncation Without Matryoshka Learning, Except In Heavy Truncation Scenarios](https://mlsanity.com/paper/2605.16608/) (2026)
- [GraSP-VL: Length as a Semantic Granularity Interface for Vision-Language Representations](https://mlsanity.com/paper/2605.17727/) (2026)
- [DIVE: Embedding Compression via Self-Limiting Gradient Updates](https://mlsanity.com/paper/2605.20689/) (2026)
- [Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings](https://mlsanity.com/paper/2609.25165/) (2026)
- [When Does Dense Retrieval Need Asymmetric Geometry? A Bias-Variance Theory of Shared and Dual Projections](https://mlsanity.com/paper/2609.32488/) (2026)

## Abstract (arXiv)

Embeddings from Large Language Models (LLMs) have emerged as critical components in various applications, particularly for information retrieval. While high-dimensional embeddings generally demonstrate superior performance as they contain more salient information, their practical application is frequently hindered by elevated computational latency and the associated higher cost. To address these challenges, we propose Matryoshka-Adaptor, a novel tuning framework designed for the customization of LLM embeddings. Matryoshka-Adaptor facilitates substantial dimensionality reduction while maintaining comparable performance levels, thereby achieving a significant enhancement in computational efficiency and cost-effectiveness. Our framework directly modifies the embeddings from pre-trained LLMs which is designed to be seamlessly integrated with any LLM architecture, encompassing those accessible exclusively through black-box APIs. Also, it exhibits efficacy in both unsupervised and supervised learning settings. A rigorous evaluation conducted across a diverse corpus of English, multilingual, and multimodal datasets consistently reveals substantial gains with Matryoshka-Adaptor. Notably, with Google and OpenAI Embedding APIs, Matryoshka-Adaptor achieves a reduction in dimensionality ranging from two- to twelve-fold without compromising performance across multiple BEIR datasets.