mlsanityData to 14 Oct 2026

Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents

arXiv 2608.11110 ↗First posted 11 Aug 2026cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

When a tool-using agent is given the same task in a different language, does it still take the same steps? Multilingual evaluation rarely asks: it compares final answers and discards the actions. Yet those actions are the product: they fix cost and latency, decide how the system fails, and are the only auditable part of its behaviour. We make the action policy the measured object across 8 models, 6 parallel benchmarks and 41 languages (2.38M rollouts). The naive measurement fails: five confounds sit between raw trace similarity and any defensible claim, each able to flip a conclusion. Short traces score higher, empty traces score perfectly, unrelated traces agree by chance over half the time, the gap is capped by each model's reproducibility, and a model asked the same question twice in one language answers differently, leaving no baseline. We remove all five, and every correction makes the effect larger. Divergence proves structural, not sampling noise: it survives greedy decoding in every cell and stays flat as temperature rises, even as models grow less self-consistent. Normalised by their own reproducibility, four very different frontier models converge under greedy decoding, each keeping 71-73% of its action policy across languages, with model identity explaining only 5.7% of the variance. Below roughly 10B parameters it breaks down, and the ordering among smaller models is largely an artifact of a chance floor we measure by permutation rather than assume. Agents route non-English tasks through English; this pivot is causally load-bearing, confirmed by a pre-registered prediction across four models, and models will not abandon it when told to. Finally, a single trace-extraction regex, not the model, manufactured a multilingual failure: two worked examples raise one model's measured accuracy twenty-sixfold while its accuracy on readable outputs barely moves.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 35 references that are in the map. First posted on arXiv 11 Aug 2026.

Builds on

30 of 35 references.

References · 30

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  8. 2025Gemma 3 Technical Reportcited by 1,824
  9. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  10. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  11. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  12. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
18 more
  1. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  2. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
  3. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  4. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  5. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  7. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  8. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  9. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  10. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  11. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  12. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  13. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  14. 2022TALM: Tool Augmented Language Modelscited by 197
  15. 2024How do Large Language Models Handle Multilingualism?cited by 150
  16. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 136
  17. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
  18. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 118

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.