# Do Code LLMs Understand Design Patterns?

arXiv: https://arxiv.org/abs/2501.04835 · HTML: https://mlsanity.com/paper/2501.04835/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 8 Jan 2025.

## Builds on (most cited first, 13 of 13)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders](https://mlsanity.com/paper/2410.20526/) (2024)
- [Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models](https://mlsanity.com/paper/2403.17359/) (2024)
- [Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Action](https://mlsanity.com/paper/2405.17822/) (2024)
- [Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?](https://mlsanity.com/paper/2410.01353/) (2024)

## Built on by (most cited first, 9 of 9)

- [MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse](https://mlsanity.com/paper/2503.18470/) (2025)
- [AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2510.01586/) (2025)
- [Using LLMs in Generating Design Rationale for Software Architecture Decisions](https://mlsanity.com/paper/2504.20781/) (2025)
- [Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action](https://mlsanity.com/paper/2506.13932/) (2025)
- [Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safety](https://mlsanity.com/paper/2508.03864/) (2025)
- [ArchAgent: Scalable Legacy Software Architecture Recovery with LLMs](https://mlsanity.com/paper/2601.13007/) (2026)
- [MetaFind: Scene-Aware 3D Asset Retrieval for Coherent Metaverse Scene Generation](https://mlsanity.com/paper/2510.04057/) (2025)
- [LLM Translation of Compiler Intermediate Representation](https://mlsanity.com/paper/2605.08247/) (2026)
- [Strategies for Guiding LLMs to Use Software Design Patterns: A Case of Singleton](https://mlsanity.com/paper/2605.26898/) (2026)

## Abstract (arXiv)

Code Large Language Models (LLMs) demonstrate great versatility in adapting to various downstream tasks, including code generation and completion, as well as bug detection and fixing. However, Code LLMs often fail to capture existing coding standards, leading to the generation of code that conflicts with the required design patterns for a given project. As a result, developers must post-process to adapt the generated code to the project's design norms. In this work, we empirically investigate the biases of Code LLMs in software development. Through carefully designed experiments, we assess the models' understanding of design patterns across recognition, comprehension, and generation. Our findings reveal that biases in Code LLMs significantly affect the reliability of downstream tasks.