# Isotropy Matters: Soft-ZCA Whitening of Embeddings for Semantic Code Search

arXiv: https://arxiv.org/abs/2411.17538 · HTML: https://mlsanity.com/paper/2411.17538/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 13 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 8 of 8)

- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [Optimal whitening and decorrelation](https://mlsanity.com/paper/1512.00809/) (2015)
- [How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy](https://mlsanity.com/paper/2109.04740/) (2021)
- [GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding](https://mlsanity.com/paper/2311.09707/) (2023)

## Built on by (most cited first, 2 of 2)

- [Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?](https://mlsanity.com/paper/2512.19115/) (2025)
- [Robust Text Watermarking for Large Language Models via Dual Semantic Embeddings](https://mlsanity.com/paper/2606.31602/) (2026)

## Abstract (arXiv)

Low isotropy in an embedding space impairs performance on tasks involving semantic inference. Our study investigates the impact of isotropy on semantic code search performance and explores post-processing techniques to mitigate this issue. We analyze various code language models, examine isotropy in their embedding spaces, and its influence on search effectiveness. We propose a modified ZCA whitening technique to control isotropy levels in embeddings. Our results demonstrate that Soft-ZCA whitening improves the performance of pre-trained code language models and can complement contrastive fine-tuning.