mlsanityData to 7 Oct 2026

Improving BM25 Code Retrieval Under Fixed Generic Tokenization: Adaptive q-Log Odds as a Drop-In BM25 Fix

arXiv 2605.18561 ↗First posted 18 May 2026cs.IR cs.AI cs.SEcited by 0

On the map: Retrieval-augmented generation → RAG for code

Abstract (arXiv)

In retrieval-augmented coding, failures often begin when the relevant file is absent from the retrieved context. Under frozen generic tokenization, where a BM25 index has been built by a search system whose analyzer the practitioner does not control, this failure is routine: BM25's logarithmic RSJ-odds IDF under-separates the identifier tail that distinguishes one function from another. We replace the outer logarithm of the Robertson-Sp"arck-Jones odds with a q-logarithm. At q=1 the transform recovers BM25 exactly by L'Hopital's rule, and for q<1 it is a Box-Cox transform of the RSJ odds with lambda = 1-q. On CoIR CodeSearchNet Go (182K documents), oracle-tuned NDCG@10 rises from 0.2575 to 0.4874 (absolute +0.2299; +89.3% relative; zero sign reversals in 10,000 paired-bootstrap resamples, reported as p <= 10^-4). The effect is graded across code languages and is near-zero on BEIR text. A one-parameter closed form estimates a corpus-level q from hapax density and stays near q=1 on corpora where BM25 is already optimal. The index-time cost is a single pass over the sparse score matrix and query latency is unchanged. A tokenizer ablation shows that identifier-aware tokenization largely removes the incremental gain from q-IDF.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in RAG for code, one of 15 directions in Retrieval-augmented generation. Builds on 7 of its 23 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

7 of 23 references.

References · 7

  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  2. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  3. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  4. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
  5. 2024BM25S: Orders of magnitude faster lexical search via eager sparse scoringcited by 87
  6. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 59
  7. 2024Long Code Arena: a Set of Benchmarks for Long-Context Code Modelscited by 48

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion
  2. 2026TurboVec: A Case Study in Cost-Efficient Private Retrieval for Enterprise RAG via Codebook-Oblivious Quantization
  3. 2025Projection and Quantisation: A Unifying View of Learning to Hash, from Random Projections to the RAG Era
  4. 2026ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval
  5. 2025CODEPROMPTZIP: Code-specific Prompt Compression for Retrieval-Augmented Generation in Coding Tasks with LMs
  6. 2024Evaluating the Effectiveness and Efficiency of Demonstration Retrievers in RAG for Coding Tasks
  7. 2026From BM25 to Corrective RAG: Benchmarking Retrieval Strategies for Text-and-Table Documents
  8. 2023BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models