mlsanityData to 7 Oct 2026

CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language

arXiv 2603.14501 ↗First posted 15 Mar 2026cs.SE cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models excel in high-resource programming languages but struggle with low-resource ones. Existing research related to low-resource programming languages primarily focuses on Domain-Specific Languages (DSLs), leaving general-purpose languages that suffer from data scarcity underexplored. To address this gap, we introduce CangjieBench, a contamination-free benchmark for Cangjie, a representative low-resource general-purpose language. The benchmark comprises 248 high-quality samples manually translated from HumanEval and ClassEval, covering both Text-to-Code and Code-to-Code tasks. We conduct a systematic evaluation of diverse LLMs under four settings: Direct Generation, Syntax-Constrained Generation, Retrieval-Augmented Generation (RAG), and Agent. Experiments reveal that Direct Generation performs poorly, whereas Syntax-Constrained Generation offers the best trade-off between accuracy and computational cost. Agent achieve state-of-the-art accuracy but incur high token consumption. Furthermore, we observe that Code-to-Code translation often underperforms Text-to-Code generation, suggesting a negative transfer phenomenon where models overfit to the source language patterns. We hope that our work will offer valuable insights into LLM generalization to unseen and low-resource programming languages. Our code and data are available at https://github.com/cjhCoder7/CangjieBench.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 15 Mar 2026.

Builds on

29 of 44 references.

References · 29

  1. 2025Qwen3 Technical Reportcited by 8,195
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  6. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  7. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
  8. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 206
  9. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  10. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 84
  11. 2024A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languagescited by 65
  12. 2021AVATAR: A Parallel Corpus for Java-Python Program Translationcited by 59
17 more
  1. 2024Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generationcited by 45
  2. 2025CLEVER: A Curated Benchmark for Formally Verified Code Generationcited by 27
  3. 2024Code Prompting Elicits Conditional Reasoning Abilities in Text+Code LLMscited by 19
  4. 2025CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilationcited by 19
  5. 2025An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunitiescited by 17
  6. 2024VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generationcited by 14
  7. 2024DocCGen: Document-based Controlled Code Generationcited by 11
  8. 2025ExeCoder: Empowering Large Language Models with Executability Representation for Code Translationcited by 10
  9. 2024ClassEval-T: Evaluating Large Language Models in Class-Level Code Translationcited by 8
  10. 2025TeXpert: A Multi-Level Benchmark for Evaluating LaTeX Code Generation by LLMscited by 6
  11. 2025Towards Realistic Project-Level Code Generation via Multi-Agent Collaboration and Semantic Architecture Modelingcited by 6
  12. 2025SolEval: Benchmarking Large Language Models for Repository-level Solidity Code Generationcited by 5
  13. 2024Do Current Language Models Support Code Intelligence for R Programming Language?cited by 4
  14. 2025Function-to-Style Guidance of LLMs for Code Translationcited by 4
  15. 2025MERA Code: A Unified Framework for Evaluating Code Generation Across Taskscited by 4
  16. 2025RepoScope: Leveraging Call Chain-Aware Multi-View Context for Repository-Level Code Generationcited by 4
  17. 2024Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarkscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
  2. 2026Rethinking Code Performance Benchmarks for LLMs
  3. 2025Holistic Evaluation of State-of-the-Art LLMs for Code Generation
  4. 2026Narrowing the Complexity Gap in the Evaluation of Large Language Models ↗
  5. 2025Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks
  6. 2025EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code
  7. 2025Evaluating Large Language Models for Functional and Maintainable Code in Industrial Settings: A Case Study at ASML
  8. 2026EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages