mlsanityData to 14 Oct 2026

BC-Bench: Evaluating Agentic Engineering in a Domain-Specific Language for ERP

arXiv 2608.20851 ↗First posted 21 Aug 2026cs.SE cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Agentic engineering systems have shown strong performance on general-purpose benchmarks, yet their effectiveness in enterprise resource planning (ERP) domain-specific languages (DSLs) remains underexplored. We introduce BC-Bench, a benchmark designed to evaluate agentic engineering on real-world tasks in AL, the DSL for Microsoft Dynamics 365 Business Central. BC-Bench comprises 101 manually curated tasks extracted from two Microsoft-owned production repositories, reflecting authentic ERP development workflows. Adapting the SWE-Bench methodology, we address the unique constraints of the AL ecosystem---including limited public resources and complex environment provisioning. Beyond generating functional code, BC-Bench evaluates test generation and supports multimodal problem statements where visual context is commonly present. We evaluate multiple frontier models across two agent harnesses, utilizing multi-run metrics to account for nondeterminism. In the Bug Fixing category, under our evaluated settings, between-model differences in resolution rate are larger than differences between the two evaluated agent harnesses, and improvements reported on general-purpose benchmarks do not consistently transfer to AL. These results highlight the need for domain-specific evaluation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 12 of its 13 references that are in the map. First posted on arXiv 21 Aug 2026.

Builds on

12 of 13 references.

References · 12

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 279
  5. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 203
  6. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  7. 2025SWE-bench Goes Live!cited by 80
  8. 2024A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languagescited by 66
  9. 2023Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMscited by 45
  10. 2024VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generationcited by 15
  11. 2025RealBench: Benchmarking Verilog Generation Models with Real-World IP Designscited by 14
  12. 2026Benchmarking Large Language Models for ABAP Code Generation: An Empirical Study on Iterative Improvement by Compiler Feedbackcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software
  2. 2025AgentArch: A Comprehensive Benchmark to Evaluate Agent Architectures in Enterprise
  3. 2025Towards a Standard, Enterprise-Relevant Agentic AI Benchmark: Lessons from 5.5 billion tokens' worth of agentic AI evaluations
  4. 2026AI-NativeBench: An Open-Source White-Box Agentic Benchmark Suite for AI-Native Systems
  5. 2026Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation
  6. 2026Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems
  7. 2025Continuous Benchmark Generation for Evaluating Enterprise-scale LLM Agents
  8. 2026ERPBench: Evaluating LLM Agents for Enterprise Decision-Making Across Competitive Market Ecologies