IndustryCode: A Benchmark for Industry Code Generation
Not on the map. Why?
Abstract (arXiv)
Code generation and comprehension by Large Language Models (LLMs) have emerged as core drivers of industrial intelligence and decision optimization, finding widespread application in fields such as finance, automation, and aerospace. Although recent advancements have demonstrated the remarkable potential of LLMs in general code generation, existing benchmarks are mainly confined to single domains and languages. Consequently, they fail to effectively evaluate the generalization capabilities required for real-world industrial applications or to reflect the coding proficiency demanded by complex industrial scenarios. To bridge this gap, we introduce IndustryCode, the first comprehensive benchmark designed to span multiple industrial domains and programming languages. IndustryCode comprises 579 sub-problems derived from 125 primary industrial challenges, accompanied by rigorous problem descriptions and test cases. It covers a wide range of fields, including finance, automation, aerospace, and remote sensing-and incorporates diverse programming languages such as MATLAB, Python, C++, and Stata. In our evaluation, the top-performing model, Claude 4.5 Opus, achieved an overall accuracy of 68.1% on sub-problems and 42.5% main problems. The benchmark dataset and automated evaluation code will be made publicly available upon acceptance.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 35 of its 49 references that are in the map. First posted on arXiv 3 Apr 2026.
Builds on
35 of 49 references.
References · 35
- 2023GPT-4 Technical Reportcited by 18,132
- 2025Qwen3 Technical Reportcited by 8,195
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023Code Llama: Open Foundation Models for Codecited by 2,105
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
- 2022Competition-Level Code Generation with AlphaCodecited by 1,427
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
- 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
- 2024Qwen2.5-Coder Technical Reportcited by 1,151
23 more
- 2025OpenAI GPT-5 System Cardcited by 910
- 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
- 2021Measuring Coding Challenge Competence With APPScited by 879
- 2023StarCoder: may the source be with you!cited by 790
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
- 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
- 2025Kimi K2: Open Agentic Intelligencecited by 523
- 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
- 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
- 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 264
- 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
- 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
- 2024SciCode: A Research Coding Benchmark Curated by Scientistscited by 172
- 2024DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?cited by 119
- 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
- 2024HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generationcited by 39
- 2023BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Modelscited by 24
- 2024LangProp: A code optimization framework using Large Language Models applied to drivingcited by 17
- 2025XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoningcited by 12
- 2025How Scientists Use Large Language Models to Programcited by 8
- 2025V-GameGym: Visual Game Generation for Code Large Language Modelscited by 8
- 2024DACO: Towards Application-Driven and Comprehensive Data Analysis via Code Generationcited by 6
- 2024Training LLMs for Generating IEC 61131-3 Structured Text with Online Feedbackcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.