SWE-Sharp-Bench: A Reproducible Benchmark for C# Software Engineering Tasks
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
AI coding agents have shown great progress on Python software engineering benchmarks like SWE-Bench, and for other languages like Java and C in benchmarks like Multi-SWE-Bench. However, C# -- a prominent enterprise language ranking #5 in the TIOBE index -- remains absent from such benchmarks. We introduce SWE-Sharp-Bench, a reproducible software engineering benchmark for C# featuring 150 instances from 17 repositories. Evaluating identical model-agent configurations across languages reveals a significant performance gap: while 70% of Python tasks in SWE-Bench Verified are solved, only 40% of our C# tasks are resolved. We open-source SWE-Sharp-Bench and our entire curation pipeline.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. First posted on arXiv 4 Nov 2025.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026SWE-rebench V2: Language-Agnostic SWE Task Collection at Scalecited by 29
- 2026REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usagecited by 5
- 2026MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Developmentcited by 0
- 2026ACE-Bench: A Lightweight Benchmark for Evaluating Azure SDK Usage Correctnesscited by 0
- 2026REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage2 Apr 2026
- 2026MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Development26 Mar 2026
- 2026SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale27 Feb 2026
- 2026ACE-Bench: A Lightweight Benchmark for Evaluating Azure SDK Usage Correctness14 Feb 2026
Similar, not cited
Close in wording, no citation either way
- 2026SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring
- 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents
- 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios
- 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
- 2025SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
- 2025Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
- 2026SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications?
- 2026SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents