Where Post-Training Quantization Breaks Text Embedders: A Measured Map Across Four Embedder Families
Not on the map. Why?
Abstract (arXiv)
Weight-only post-training quantization is the cheapest way to shrink a retrieval embedder, and the received advice for applying it -- protect the embedding table, allocate bits by module sensitivity, prefer a ranking-aware objective over weight reconstruction -- was carried into LLM quantization largely intact. We test that advice on retrieval embedders directly, quantizing five checkpoints from four architecture families across a grid of bit widths and group sizes, and isolating the embedding, attention and feed-forward blocks at each width. Every heuristic fails to transfer as stated. The embedding table never emerges as the dominant isolated protection priority in any family, despite being the largest tensor in several of them. Module sensitivity does not survive as a transferable ordering: at INT4/g16 the spread between modules is too small to allocate against, at INT3 the ordering becomes family-dependent and joint damage stops being the sum of its parts, and at INT2 comparable reconstruction error accompanies retention ranging from 1.3 to 65.9 percent of full precision. A cheap reconstruction proxy is useful for screening uniform bit widths but substantially less reliable for choosing which tensors to protect; its apparent strength across the whole grid is a range-extension artifact. A distilled 109M student at INT3 holds 78.04 NDCG@10 in 68.4 MB and dominates the extreme-PTQ arm of its own 0.6B teacher, 297.9 MB at 64.46, on both size and quality -- but only inside the task it was distilled for. Sizes are byte counts of files that exist rather than arithmetic estimates, and the measurement repository carries the byte provenance for every one of them.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 45 references that are in the map. First posted on arXiv 14 Sep 2026.
Builds on
39 of 45 references.
References · 39
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
- 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
- 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
- 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
- 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
- 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
- 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
- 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
27 more
- 2024SpinQuant: LLM quantization with learned rotationscited by 471
- 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
- 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
- 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
- 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
- 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
- 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
- 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
- 2019Q8BERT: Quantized 8Bit BERTcited by 300
- 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
- 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 261
- 2021I-BERT: Integer-only BERT Quantizationcited by 255
- 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
- 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
- 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
- 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
- 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 139
- 2019Loss Aware Post-training Quantizationcited by 97
- 2024T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edgecited by 55
- 2021Efficient Passage Retrieval with Hashing for Open-domain Question Answeringcited by 53
- 2026SkillRet: A Large-Scale Benchmark for Skill Retrieval in LLM Agentscited by 39
- 2021Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performancecited by 35
- 2021Learning Discrete Representations via Constrained Clustering for Effective and Efficient Dense Retrievalcited by 27
- 2026What Limits Does Quantization Place on Dense Top-k Retrieval? A Theoretical Studycited by 4
- 2022Compact Token Representations with Contextual Quantization for Efficient Document Re-rankingcited by 3
- 2026LLMs Meet Isolation Kernel: Lightweight, Learning-free Binary Embeddings for Fast Retrievalcited by 3
- 2026When Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compressioncited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Quantize by Drift: Label-Free Mixed-Precision Post-Training Quantization for Text Embedders
- 2026BitNet Text Embeddings
- 2025Projection and Quantisation: A Unifying View of Learning to Hash, from Random Projections to the RAG Era
- 2026MRVQ: One Resident Index for Dimension- and Rate-Elastic Vector Search
- 2026On the Spectral Flattening of Quantized Embeddings
- 2025Exploring Layer-wise Information Effectiveness for Post-Training Quantization in Small Language Models
- 2026FastE: Readout-Triggered Token Compression for LLM Embedding Inference
- 2024Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels