# Codex Hacks HackerRank: Memorization Issues and a Framework for Code Synthesis Evaluation

arXiv: https://arxiv.org/abs/2212.02684 · HTML: https://mlsanity.com/paper/2212.02684/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 6 Dec 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [A Survey of Machine Learning for Big Code and Naturalness](https://mlsanity.com/paper/1709.06182/) (2017)
- [Synchromesh: Reliable code generation from pre-trained language models](https://mlsanity.com/paper/2201.11227/) (2022)
- [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018)
- [Neuro-Symbolic Program Synthesis](https://mlsanity.com/paper/1611.01855/) (2016)
- [What do pre-trained code models know about code?](https://mlsanity.com/paper/2108.11308/) (2021)
- [Automatic Program Repair with OpenAI's Codex: Evaluating QuixBugs](https://mlsanity.com/paper/2111.03922/) (2021)
- [SourceFinder: Finding Malware Source-Code from Publicly Available Repositories](https://mlsanity.com/paper/2005.14311/) (2020)
- [Solving Linear Algebra by Program Synthesis](https://mlsanity.com/paper/2111.08171/) (2021)
- [Pop Quiz! Can a Large Language Model Help With Reverse Engineering?](https://mlsanity.com/paper/2202.01142/) (2022)
- [Solving Probability and Statistics Problems by Program Synthesis](https://mlsanity.com/paper/2111.08267/) (2021)

## Built on by (most cited first, 14 of 14)

- [Large Language Models and Simple, Stupid Bugs](https://mlsanity.com/paper/2303.11455/) (2023)
- [Data Contamination Through the Lens of Time](https://mlsanity.com/paper/2310.10628/) (2023)
- [Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses](https://mlsanity.com/paper/2306.10073/) (2023)
- [Large Language Models (GPT) Struggle to Answer Multiple-Choice Questions about Code](https://mlsanity.com/paper/2303.08033/) (2023)
- [Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?](https://mlsanity.com/paper/2306.13906/) (2023)
- [Traces of Memorisation in Large Language Models for Code](https://mlsanity.com/paper/2312.11658/) (2023)
- [Exploring the Robustness of Large Language Models for Solving Programming Problems](https://mlsanity.com/paper/2306.14583/) (2023)
- [Chatbots As Fluent Polyglots: Revisiting Breakthrough Code Snippets](https://mlsanity.com/paper/2301.03373/) (2023)
- [Evaluating GPT's Programming Capability through CodeWars' Katas](https://mlsanity.com/paper/2306.01784/) (2023)
- [INSPECT: Intrinsic and Systematic Probing Evaluation for Code Transformers](https://mlsanity.com/paper/2312.05092/) (2023)
- [Can Generative Pre-trained Transformers (GPT) Pass Assessments in Higher Education Programming Courses?](https://mlsanity.com/paper/2303.09325/) (2023)
- [Breaking the Silence: the Threats of Using LLMs in Software Engineering](https://mlsanity.com/paper/2312.08055/) (2023)
- [RETROcode: Leveraging a Code Database for Improved Natural Language to Code Generation](https://mlsanity.com/paper/2504.05759/) (2025)
- [How Do Semantically Equivalent Code Transformations Impact Membership Inference on LLMs for Code?](https://mlsanity.com/paper/2512.15468/) (2025)

## Abstract (arXiv)

The Codex model has demonstrated extraordinary competence in synthesizing code from natural language problem descriptions. However, in order to reveal unknown failure modes and hidden biases, such large-scale models must be systematically subjected to multiple and diverse evaluation studies. In this work, we evaluate the code synthesis capabilities of the Codex model based on a set of 115 Python problem statements from a popular competitive programming portal: HackerRank. Our evaluation shows that Codex is indeed proficient in Python, solving 96% of the problems in a zero-shot setting, and 100% of the problems in a few-shot setting. However, Codex exhibits clear signs of generating memorized code based on our evaluation. This is alarming, especially since the adoption and use of such models could directly impact how code is written and produced in the foreseeable future. With this in mind, we further discuss and highlight some of the prominent risks associated with large-scale models of source code. Finally, we propose a framework for code-synthesis evaluation using variations of problem statements based on mutations.