Beyond Retrieval: A Multitask Benchmark and Model for Code Search

Code search has usually been evaluated as first-stage retrieval, even though production systems rely on broader pipelines with reranking and developer-style queries. Existing benchmarks also suffer from data contamination, label noise, and degenerate binary relevance. In this paper, we introduce textscCoREB, a contamination-limited, multitask underlinecode underlineretrieval and runderlineeranking underlinebenchmark, together with a fine-tuned code reranker, that goes beyond retrieval to cover the full code search pipeline. textscCoREB is built from counterfactually rewritten LiveCodeBench problems in five programming languages and delivered as timed releases with graded relevance judgments. We benchmark eleven embedding models and five rerankers across three tasks: text-to-code, code-to-text, and code-to-code. Our experiments reveal that: circone code-specialised embeddings dominate code-to-code retrieval (~2× over general encoders), yet no single model wins all three tasks; circtwo short keyword queries, the format closest to real developer search, collapse every model to near-zero nDCG@10; circthree off-the-shelf rerankers are task-asymmetric, with a 12-point swing on code-to-code and no baseline net-positive across all tasks; circfour our fine-tuned textscCoREB-Reranker is the first to achieve consistent gains across all three tasks. The data and model are released.