mlsanityData to 7 Oct 2026

Do programming languages still matter to your AI coding agent teammate? Evidence at scale from chess engines

arXiv 2606.13763 ↗First posted 11 Jun 2026cs.SEcited by 1

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Frontier coding agents now promise end-to-end authorship of complete software systems. Two empirical questions follow: can AI coding-agent teammates program in any target language, including ones with no comparable prior open-source artefact? If so, does language choice still shape the artefact, and along which dimensions? We study both through a polyglot case study built around chess engines: non-trivial multi-component systems that admit a hierarchy of language-agnostic oracles, from exact move-generation correctness to a strength scale (Elo), observable from Rust to Brainfuck. We prompted two frontier agents (Claude Code and Codex) at the capability level, without chess knowledge or implementation guidance, under a documented intervention and stopping policy. The agents produced 34 chess engines spanning 17 primary programming languages, from mainstream to specialised, domain-specific, legacy, and esoteric targets. We combine per-engine feature analysis, independent Elo assessment, and session trajectories with qualitative analysis of code and transcripts. Frontier coding agents are genuinely polyglot: every language we tried produced at least one feature-rich working engine, several with no prior open-source counterpart of comparable scope (e.g., LaTeX), and the code is synthesised from scratch rather than copied. Yet language choice still matters: strong playing strength is only reachable in mainstream compiled languages, cost and engineering effort grow sharply as the language becomes more exotic, and feature choices shift across language families. Agents validate their own work unprompted, but their strength self-estimates are biased and a few engines cheated by calling a chess library. Programming language is no longer about whether AI teammates can build a working system, but about performance, cost, what gets built, and how much human supervision validation still needs.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 47 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

20 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025The Rise of AI Teammates in Software Engineering (SE) 3.0: How Autonomous Coding Agents Are Reshaping Software Engineering20 Jul
  2. 2025Unified Software Engineering Agent as AI Software Engineer17 Jun
  3. 2025SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents26 May
  4. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents11 Apr
  5. 2024Agentless: Demystifying LLM-based Software Engineering Agents1 Jul

Other references · 15

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  5. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  6. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  7. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  8. 2025Measuring AI Ability to Complete Long Software Taskscited by 142
  9. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  10. 2024Commit0: Library Generation from Scratchcited by 50
  11. 2026ProgramBench: Can Language Models Rebuild Programs From Scratch?cited by 47
  12. 2024Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Studycited by 40
3 more
  1. 2025Guidelines for Empirical Studies in Software Engineering involving Large Language Modelscited by 18
  2. 2026Stalled, Biased, and Confused: Uncovering Reasoning Failures in LLMs for Cloud-Based Root Cause Analysiscited by 8
  3. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages
  2. 2026Latent Programming Horizons in Coding Agents
  3. 2026Up and Down the Abstraction Ladder: Code-Based Skills for Language Agents
  4. 2026Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages
  5. 2026OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents
  6. 2026How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests
  7. 2025FreshBrew: A Benchmark for Evaluating AI Agents on Java Code Migration
  8. 2026Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality