mlsanityData to 14 Oct 2026

Evaluating SAP Joule for Code Generation

arXiv 2509.24828 ↗First posted 29 Sep 2025cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

SAP has released its own proprietary generative model SAP Joule, intended for various generative tasks, including serving as a code assistant for software engineers. While Joule is yet not focused on SAP-specific ABAP code generation, it can be used for other common languages, including Javascript. This paper compares SAP Joules Javascript coding capabilities against a total of 29 other models using the HumanEval-X Javascript benchmark. SAP Joule achieves a strict accuracy of 80.49% as the fifth best model in our evaluation. To the best of our knowledge, this is the first comparative evaluation of SAP Joule code generation capabilities.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 10 of its 31 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

10 of 31 references.

References · 10

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  3. 2021Measuring Coding Challenge Competence With APPScited by 902
  4. 2024A Survey on Large Language Models for Code Generationcited by 658
  5. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  6. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  7. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  8. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 189
  9. 2022Multi-lingual Evaluation of Code Generation Modelscited by 159
  10. 2023xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrievalcited by 47

Built on by

Not cited yet.