# On the Effectiveness of Large Language Models in Domain-Specific Code Generation

arXiv: https://arxiv.org/abs/2312.01639 · HTML: https://mlsanity.com/paper/2312.01639/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2023.

## Built on by (most cited first, 30 of 44)

- [On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub](https://mlsanity.com/paper/2509.14745/) (2025)
- [LLaMoCo: Instruction Tuning of Large Language Models for Optimization Code Generation](https://mlsanity.com/paper/2403.01131/) (2024)
- [In Line with Context: Repository-Level Code Generation via Context Inlining](https://mlsanity.com/paper/2601.00376/) (2026)
- [Retrieval-Augmented Data Augmentation for Low-Resource Domain Tasks](https://mlsanity.com/paper/2402.13482/) (2024)
- [CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding](https://mlsanity.com/paper/2512.04111/) (2025)
- [Towards Formal Verification of LLM-Generated Code from Natural Language Prompts](https://mlsanity.com/paper/2507.13290/) (2025)
- [GPT-4.1 Sets the Standard in Automated Experiment Design Using Novel Python Libraries](https://mlsanity.com/paper/2508.00033/) (2025)
- [ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Design](https://mlsanity.com/paper/2601.21448/) (2026)
- [LLM4DS: Evaluating Large Language Models for Data Science Code Generation](https://mlsanity.com/paper/2411.11908/) (2024)
- [Man-Made Heuristics Are Dead. Long Live Code Generators!](https://mlsanity.com/paper/2510.08803/) (2025)
- [Can ChatGPT implement finite element models for geotechnical engineering applications?](https://mlsanity.com/paper/2501.02199/) (2025)
- [GeoCAD: Local Geometry-Controllable CAD Generation with Large Language Models](https://mlsanity.com/paper/2506.10337/) (2025)
- [KOCO-BENCH: Can Large Language Models Leverage Domain Knowledge in Software Development?](https://mlsanity.com/paper/2601.13240/) (2026)
- [ModiGen: A Large Language Model-Based Workflow for Multi-Task Modelica Code Generation](https://mlsanity.com/paper/2503.18460/) (2025)
- [Optimizing Retrieval Augmented Generation for Object Constraint Language](https://mlsanity.com/paper/2505.13129/) (2025)
- [BioMol-MQA: A Multi-Modal Question Answering Dataset For LLM Reasoning Over Bio-Molecular Interactions](https://mlsanity.com/paper/2506.05766/) (2025)
- [The Impact of LLM-Assistants on Software Developer Productivity: A Systematic Review and Mapping Study](https://mlsanity.com/paper/2507.03156/) (2025)
- [Exploring Data-Efficient Adaptation of Large Language Models for Code Generation](https://mlsanity.com/paper/2403.00046/) (2024)
- [GEE-OPs: An Operator Knowledge Base for Geospatial Code Generation on the Google Earth Engine Platform Powered by Large Language Models](https://mlsanity.com/paper/2412.05587/) (2024)
- [Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Prompting](https://mlsanity.com/paper/2509.06770/) (2025)
- [Repo0: Design-Driven Zero-to-All Code Generation](https://mlsanity.com/paper/2608.19854/) (2026)
- [OnlySportsLM: Optimizing Sports-Domain Language Models with SOTA Performance under Billion Parameters](https://mlsanity.com/paper/2409.00286/) (2024)
- [Deep Learning-based Intrusion Detection Systems: A Survey](https://mlsanity.com/paper/2504.07839/) (2025)
- [FineScope : SAE-guided Data Selection Enables Domain Specific LLM Pruning and Finetuning](https://mlsanity.com/paper/2505.00624/) (2025)
- [CelloAI: Leveraging Large Language Models for HPC Software Development in High Energy Physics](https://mlsanity.com/paper/2508.16713/) (2025)
- [Learning from Execution: Self-Evolving Memory for Private-Library Code Generation](https://mlsanity.com/paper/2604.24222/) (2026)
- [From PowerPoint UI Sketches to Web-Based Applications: Pattern-Driven Code Generation for GIS Dashboard Development Using Knowledge-Augmented LLMs, Context-Aware Visual Prompting, and the React Framework](https://mlsanity.com/paper/2502.08756/) (2025)
- [A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standards](https://mlsanity.com/paper/2505.13766/) (2025)
- [Real-Time World Crafting: Generating Structured Game Behaviors from Natural Language with Large Language Models](https://mlsanity.com/paper/2510.16952/) (2025)
- [An Agentic Framework for Neuro-Symbolic Programming](https://mlsanity.com/paper/2601.00743/) (2026)

## Abstract (arXiv)

Large language models (LLMs) such as ChatGPT have shown remarkable capabilities in code generation. Despite significant achievements, they rely on enormous training data to acquire a broad spectrum of open-domain knowledge. Besides, their evaluation revolves around open-domain benchmarks like HumanEval, which primarily consist of programming contests. Therefore, it is hard to fully characterize the intricacies and challenges associated with particular domains (e.g., web, game, and math). In this paper, we conduct an in-depth study of the LLMs in domain-specific code generation. Our results demonstrate that LLMs exhibit sub-optimal performance in generating domain-specific code, due to their limited proficiency in utilizing domain-specific libraries. We further observe that incorporating API knowledge as prompts can empower LLMs to generate more professional code. Based on these findings, we further investigate how to effectively incorporate API knowledge into the code generation process. We experiment with three strategies for incorporating domain knowledge, namely, external knowledge inquirer, chain-of-thought prompting, and chain-of-thought fine-tuning. We refer to these strategies as a new code generation approach called DomCoder. Experimental results show that all strategies of DomCoder lead to improvement in the effectiveness of domain-specific code generation under certain settings.