mlsanityData to 7 Oct 2026

Terminal-Bench-LILT: Multilingual Agentic Coding Benchmark Grounded in Language, Region, and Culture

arXiv 2608.28641 ↗First posted 11 Aug 2026cs.CL cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Most evaluations for coding agents are conducted exclusively in English, which does not reflect real-world multilingual deployment. We present Terminal-Bench-LILT, a suite of 300 authentic coding tasks in ten languages: Arabic, Czech, German, Spanish, Hindi, Japanese, Korean, Serbian, Turkish, and Chinese. Each task targets issues specific to non-English software development that have no direct English equivalent, e.g., internationalization, encoding, text normalization, and cultural conventions. All tasks are authored by native-speaker programmers and validated through a multi-stage quality control pipeline. Evaluation of six frontier models reveals that even the strongest model reaches only 63.1% pass rate, with many tasks unsolved by any model. Performance varies substantially by language and does not track general coding benchmark rankings, highlighting that multilingual coding competence is a distinct and underexplored capability axis. Sample tasks are available at https://github.com/lilt/terminal-bench-lilt

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 16 of its 34 references that are in the map. First posted on arXiv 11 Aug 2026.

Builds on

16 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025MAPS: A Multilingual Benchmark for Agent Performance and Security21 May

Other references · 15

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  4. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
  5. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 484
  6. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  7. 2024BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languagescited by 171
  8. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  9. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  10. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  11. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 84
  12. 2025Do Multilingual LLMs Think In English?cited by 75
3 more
  1. 2025Long Chain-of-Thought Reasoning Across Languagescited by 18
  2. 2024Exploring Multi-Lingual Bias of Large Code Models in Code Generationcited by 17
  3. 2026"Write in English, Nobody Understands Your Language Here": A Study of Non-English Trends in Open-Source Repositoriescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
  2. 2026Cross-Lingual Token Arbitrage: Optimizing Code Agent Context Windows via Local LLM Preprocessing
  3. 2026RuBench: A Repository-Level Agentic Coding Benchmark with Natively Authored Russian Task Specifications
  4. 2026GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translation
  5. 2024Large Language Model Evaluation Via Multi AI Agents: Preliminary results
  6. 2026BabelArena: A Large-Scale Multilingual Benchmark for LLM Agents
  7. 2026WorldBench: Culturally Grounded Benchmark for Multilingual Agents
  8. 2026Hybrid-Gym: Training Coding Agents to Generalize Across Tasks