# AlgoSimBench: Identifying Algorithmically Similar Problems for Competitive Programming

arXiv: https://arxiv.org/abs/2507.15378 · HTML: https://mlsanity.com/paper/2507.15378/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 35 of its 57 references that are in the map.
- First posted on arXiv 21 Jul 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Distributed Representations of Sentences and Documents](https://mlsanity.com/paper/1405.4053/) (2014)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [Precise Zero-Shot Dense Retrieval without Relevance Labels](https://mlsanity.com/paper/2212.10496/) (2022)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2401.03065/) (2024)
- [MapCoder: Multi-Agent Code Generation for Competitive Problem Solving](https://mlsanity.com/paper/2405.11403/) (2024)
- [Natural Language to Code Translation with Execution](https://mlsanity.com/paper/2204.11454/) (2022)
- [Learning Performance-Improving Code Edits](https://mlsanity.com/paper/2302.07867/) (2023)
- [Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering](https://mlsanity.com/paper/2401.08500/) (2024)
- [CodeRAG-Bench: Can Retrieval Augment Code Generation?](https://mlsanity.com/paper/2406.14497/) (2024)
- [Contrastive Code Representation Learning](https://mlsanity.com/paper/2007.04973/) (2020)
- [Can Language Models Solve Olympiad Programming?](https://mlsanity.com/paper/2404.10952/) (2024)
- [Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning](https://mlsanity.com/paper/2305.17256/) (2023)
- [CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking](https://mlsanity.com/paper/2412.01007/) (2024)
- [CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models](https://mlsanity.com/paper/2411.04329/) (2024)
- [CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval](https://mlsanity.com/paper/2411.12644/) (2024)
- [ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?](https://mlsanity.com/paper/2407.14044/) (2024)
- [NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness](https://mlsanity.com/paper/2401.15963/) (2024)
- [Code Representation Learning At Scale](https://mlsanity.com/paper/2402.01935/) (2024)
- [Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities](https://mlsanity.com/paper/2406.02787/) (2024)

## Built on by (most cited first, 2 of 2)

- [CP-Agent: A Calibrated Risk-Controlled Agent for Feedback-Driven Competitive Programming](https://mlsanity.com/paper/2605.24693/) (2026)
- [RoMathExam: A Longitudinal Dataset of Romanian Math Exams (1895-2025) with a Seven-Decade Core (1957-2025)](https://mlsanity.com/paper/2604.16392/) (2026)

## Abstract (arXiv)

Recent reasoning-enhanced Large Language Models (LLMs) have achieved promising results in solving complex competitive programming problems. However, it remains unclear whether these reasoning abilities generalize to relevant tasks, like identifying algorithmically similar problems (ASPs). We introduce AlgoSimBench, a benchmark of 402 multiple-choice questions curated in an adversarial setting: each given reference problem is paired with one algorithmically similar problem and three distractors that are semantically close but algorithmically dissimilar. This design forces models to rely on algorithmic reasoning rather than superficial textual cues. Our evaluation shows that LLMs consistently struggle under this setting. To address this gap, we propose Attempted Solution Matching (ASM), which leverages LLM-generated solution attempts to assess similarity, yielding an average accuracy improvement of 9% across models. Beyond LLM evaluation, AlgoSimBench also probes code retrieval methods; when combined with BM25, ASM achieves an additional 11.8% gain over state-of-the-art embedding models. AlgoSimBench offers a challenging testbed that facilitates future studies on LLMs and retrieval methods.