# Matryoshka Hash Representations for Model-Aware Compact Semantic Retrieval

arXiv: https://arxiv.org/abs/2609.07276 · HTML: https://mlsanity.com/paper/2609.07276/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 31 of its 50 references that are in the map.
- First posted on arXiv 7 Sep 2026.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 30 of 31)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [C-Pack: Packed Resources For General Chinese Embeddings](https://mlsanity.com/paper/2309.07597/) (2023)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims](https://mlsanity.com/paper/2012.00614/) (2020)
- [HashNet: Deep Learning to Hash by Continuation](https://mlsanity.com/paper/1702.00758/) (2017)
- [Efficient Passage Retrieval with Hashing for Open-domain Question Answering](https://mlsanity.com/paper/2106.00882/) (2021)
- [Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance](https://mlsanity.com/paper/2108.00644/) (2021)
- [Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrieval](https://mlsanity.com/paper/2110.05789/) (2021)
- [Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions](https://mlsanity.com/paper/2407.20243/) (2024)
- [SMEC: Rethinking Matryoshka Representation Learning for Retrieval Embedding Compression](https://mlsanity.com/paper/2510.12474/) (2025)
- [Collaborative Learning for Extremely Low Bit Asymmetric Hashing](https://mlsanity.com/paper/1809.09329/) (2018)
- [Binary Embedding-based Retrieval at Tencent](https://mlsanity.com/paper/2302.08714/) (2023)
- [Injecting Domain Adaptation with Learning-to-hash for Effective and Efficient Zero-shot Dense Retrieval](https://mlsanity.com/paper/2205.11498/) (2022)
- [LEANN: A Low-Storage Vector Index](https://mlsanity.com/paper/2506.08276/) (2025)
- [Refining BERT Embeddings for Document Hashing via Mutual Information Maximization](https://mlsanity.com/paper/2109.02867/) (2021)
- [LLMs Meet Isolation Kernel: Lightweight, Learning-free Binary Embeddings for Fast Retrieval](https://mlsanity.com/paper/2601.09159/) (2026)
- [Nested Hash Layer: A Plug-and-play Module for Multiple-length Hash Code Learning](https://mlsanity.com/paper/2412.08922/) (2024)
- [More Than Efficiency: Embedding Compression Improves Domain Adaptation in Dense Retrieval](https://mlsanity.com/paper/2601.13525/) (2026)
- [MESS: Fast and Private Semantic Search on Multi-Graph HNSW](https://mlsanity.com/paper/2607.28999/) (2026)
- [Spruce: Scalable Private Outsourced Retrieval Using Compact Embeddings](https://mlsanity.com/paper/2609.03376/) (2026)

## Built on by (most cited first, 1 of 1)

- [Semantics Delivery Network: Rethinking Web Retrieval Infrastructure for LLM Agents](https://mlsanity.com/paper/2609.22486/) (2026)

## Abstract (arXiv)

Retrieval-augmented generation (RAG) depends on dense retrieval: each document is stored as a learned vector, and a query is answered by finding its nearest neighbors in that vector space. Keeping one full-precision vector per document is the dominant index cost at corpus scale, so retrieval systems replace each vector with a short code of a few bytes---a step called quantization. Standard quantizers such as product quantization (PQ) pick the code that reconstructs the original vector most closely. A single code is even more useful if it serves several byte budgets at once: when its short prefixes are each directly searchable, a deployment can set its efficiency--quality operating point without re-encoding the corpus. But training all prefixes under one objective makes the early bits a compromise across budgets---short codes improve while the full-width code degrades. Quantization to low-bit representation, such as binary codes, further sharpens the conflict. We introduce Matryoshka Hash Representations (MHR), a two-stage procedure that separates full-width training from prefix organization. MHR first learns a longer binary code, then freezes the model and trains additional zero-initialized residual code adaptors for directly searchable prefixes. Documents are stored at one bit per coordinate, while queries keep continuous logits like PQ to attain sufficient expressivity. We implement the search process with FAISS FastScan. Trained on MS MARCO and zero-shot transferred to seven BEIR datasets, MHR reaches .5561 NDCG@10 and .6535 Recall@100 at 32 bytes, surpassing the best baseline of the same budget. The advantage is more pronounced in lower budgets. The same code also strengthens two common pipelines: shortlisting candidates for full-precision reranking, and pruning a low-storage graph index such as LEANN.