mlsanityData to 14 Oct 2026

LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs

arXiv 2504.14655 ↗First posted 20 Apr 2025cs.LG cs.CL cs.SEcited by 70

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

We introduce LeetCodeDataset, a high-quality benchmark for evaluating and training code-generation models, addressing two key challenges in LLM research: the lack of reasoning-focused coding benchmarks and self-contained training testbeds. By curating LeetCode Python problems with rich metadata, broad coverage, 100+ test cases per problem, and temporal splits (pre/post July 2024), our dataset enables contamination-free evaluation and efficient supervised fine-tuning (SFT). Experiments show reasoning models significantly outperform non-reasoning counterparts, while SFT with only 2.6K model-generated solutions achieves performance comparable to 110K-sample counterparts. The dataset and evaluation framework are available on Hugging Face and Github.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 18 of its 24 references that are in the map. First posted on arXiv 20 Apr 2025.

Builds on

18 of 24 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings2 Jan
  2. 2024Qwen2.5-Coder Technical Report18 Sep
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar
  4. 2022Competition-Level Code Generation with AlphaCode8 Feb

Other references · 14

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  7. 2021Measuring Coding Challenge Competence With APPScited by 902
  8. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  9. 2023TACO: Topics in Algorithmic COde generation datasetcited by 160
  10. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 136
  11. 2025Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedbackcited by 26
  12. 2025AdaptiveStep: Automatically Dividing Reasoning Step through Model Confidencecited by 24
2 more
  1. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 22
  2. 2024Leveraging Web-Crawled Data for High-Quality Fine-Tuningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 27; 2026: 43.

70 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025Skywork Open Reasoner 1 Technical Reportcited by 165
  3. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspectivecited by 76
  4. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
  5. 2025Pruning the Unsurprising: Efficient LLM Reasoning via First-Token Surprisalcited by 32
  6. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  7. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 30
  8. 2025CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignmentcited by 24
  9. 2025FrontierCS: Evolving Challenges for Evolving Intelligencecited by 23
  10. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 21
  11. 2025Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learningcited by 17
  12. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVRcited by 17
38 more
  1. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  2. 2025Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifierscited by 14
  3. 2025BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuningcited by 14
  4. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimizationcited by 11
  5. 2026Goedel-Code-Prover: Hierarchical Proof Search for Open State-of-the-Art Code Verificationcited by 11
  6. 2025Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learningcited by 9
  7. 2026Expanding LLM Agent Boundaries with Strategy-Guided Explorationcited by 9
  8. 2026Loop the Loopies!cited by 8
  9. 2025VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable Codecited by 7
  10. 2025Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functionscited by 6
  11. 2025Local Success Does Not Compose: Benchmarking Large Language Models for Compositional Formal Verificationcited by 6
  12. 2026Beyond Binary: Turning Partial Success into Dense Verifiable Rewards for Reinforcement Learning in Code Generationcited by 5
  13. 2026FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoningcited by 5
  14. 2026Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVRcited by 5
  15. 2025Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcyclingcited by 4
  16. 2025How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspectivecited by 4
  17. 2026CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generationcited by 4
  18. 2025Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Codecited by 3
  19. 2026Scaling Small Agents Through Strategy Auctionscited by 3
  20. 2026A Theoretical Analysis of Test-Driven Code Generationcited by 3
  21. 2025HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learningcited by 2
  22. 2026ACES: Who Tests the Tests? Leave-One-Out AUC Consistency for Code Generationcited by 2
  23. 2026QiMeng-PRepair: Precise Code Repair via Edit-Aware Reward Optimizationcited by 2
  24. 2025LLM Performance for Code Generation on Noisy Taskscited by 1
  25. 2025Small Edits, Big Consequences: Telling Good from Bad Robustness in Large Language Modelscited by 1
  26. 2025Calibrated Reasoning: An Explanatory Verifier for Dynamic and Efficient Problem-Solvingcited by 1
  27. 2026InfoSynth: Information-Guided Benchmark Synthesis for LLMscited by 1
  28. 2026AlgBench: To What Extent Do Large Reasoning Models Understand Algorithms?cited by 1
  29. 2026RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Modelscited by 1
  30. 2026Think Anywhere in Code Generationcited by 1
  31. 2026Steerable Instruction Following Coding Data Synthesis with Actor-Parametric Schema Co-Evolutioncited by 1
  32. 2026Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Training-Time Reward Hacking in Code Generationcited by 1
  33. 2026Characterizing Readability Issue Patterns and the Role of Prompt Design in LLM-Generated Codecited by 1
  34. 2026Enhancing the Code Reasoning Capabilities of LLMs via Consistency-based Reinforcement Learningcited by 1
  35. 2026Verifiable Environments Are LEGO Bricks: Recursive Composition for Reasoning Generalizationcited by 1
  36. 2026Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learningcited by 1
  37. 2026DiDPO: Diff-in-Diff Policy Optimization for Coding Agent Trainingcited by 1
  38. 2026RLTL;DR: Self-improvement by Internalizing Self-generated Feedbackcited by 1

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
  2. 2026Evaluating LLMs Code Reasoning Under Real-World Context
  3. 2024CodeLutra: Boosting LLM Code Generation via Preference-Guided Refinement
  4. 2025Rethinking Verification for LLM Code Generation: From Generation to Testing
  5. 2025Increasing LLM Coding Capabilities through Diverse Synthetic Coding Tasks
  6. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
  7. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset
  8. 2026Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks