CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming
Not on the map. Why?
Abstract (arXiv)
Competitive programming benchmarks are widely used in scenarios such as programming contests and large language model assessments. However, the growing presence of duplicate or highly similar problems raises concerns not only about competition fairness, but also about the validity of competitive programming as a benchmark for model evaluation. In this paper, we propose a new problem, similar question retrieval, to tackle this issue. Due to the lack of both data and models, solving this problem is challenging. To this end, we introduce CPRet, a retrieval-oriented benchmark suite for competitive programming, covering four retrieval tasks: two code-centric (i.e., Text-to-Code, Code-to-Code) and two newly proposed problem-centric tasks (i.e., Problem-to-Duplicate, Simplified-to-Full) built from a combination of automatically crawled problem-solution data and manually curated annotations. Our contribution includes both high-quality training data and temporally separated test sets for reliable evaluation. Besides, we further develop two task-specialized retrievers based on this dataset: CPRetriever-Code, trained with a novel Group-InfoNCE loss for problem-code alignment, and CPRetriever-Prob, fine-tuned for identifying problem-level similarity. Both models achieve strong results and are open-sourced for local use. Finally, we analyze LiveCodeBench and find that high-similarity problems inflate model pass rates and reduce differentiation, underscoring the need for similarity-aware evaluation in future benchmarks. Github: https://github.com/coldchair/CPRet Online Demo: https://www.cpret.online/
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 45 references that are in the map. First posted on arXiv 19 May 2025.
Builds on
30 of 45 references.
References · 30
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
- 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
- 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
- 2022Competition-Level Code Generation with AlphaCodecited by 1,462
- 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
- 2021Measuring Coding Challenge Competence With APPScited by 902
18 more
- 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
- 2022MTEB: Massive Text Embedding Benchmarkcited by 640
- 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
- 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 372
- 2023Improving Text Embeddings with Large Language Modelscited by 303
- 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
- 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solvingcited by 177
- 2023TACO: Topics in Algorithmic COde generation datasetcited by 160
- 2021Scaling Deep Contrastive Learning Batch Size under Memory Limited Setupcited by 127
- 2024Jasper and Stella: distillation of SOTA embedding modelscited by 112
- 2024NV-Retriever: Improving text embedding models with effective hard-negative miningcited by 84
- 2024Can Language Models Solve Olympiad Programming?cited by 72
- 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 57
- 2024CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrievalcited by 34
- 2024Code Representation Learning At Scalecited by 26
- 2024Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Taskscited by 22
- 2023Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embeddingcited by 9
- 2023Duplicate Question Retrieval and Confirmation Time Prediction in Software Communitiescited by 5
Built on by
Citing papers per year: 2026: 3.
3 papers.