Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents
Not on the map. Why?
Abstract (arXiv)
When a tool-using agent is given the same task in a different language, does it still take the same steps? Multilingual evaluation rarely asks: it compares final answers and discards the actions. Yet those actions are the product: they fix cost and latency, decide how the system fails, and are the only auditable part of its behaviour. We make the action policy the measured object across 8 models, 6 parallel benchmarks and 41 languages (2.38M rollouts). The naive measurement fails: five confounds sit between raw trace similarity and any defensible claim, each able to flip a conclusion. Short traces score higher, empty traces score perfectly, unrelated traces agree by chance over half the time, the gap is capped by each model's reproducibility, and a model asked the same question twice in one language answers differently, leaving no baseline. We remove all five, and every correction makes the effect larger. Divergence proves structural, not sampling noise: it survives greedy decoding in every cell and stays flat as temperature rises, even as models grow less self-consistent. Normalised by their own reproducibility, four very different frontier models converge under greedy decoding, each keeping 71-73% of its action policy across languages, with model identity explaining only 5.7% of the variance. Below roughly 10B parameters it breaks down, and the ordering among smaller models is largely an artifact of a chance floor we measure by permutation rather than assume. Agents route non-English tasks through English; this pivot is causally load-bearing, confirmed by a pre-registered prediction across four models, and models will not abandon it when told to. Finally, a single trace-extraction regex, not the model, manufactured a multilingual failure: two worked examples raise one model's measured accuracy twenty-sixfold while its accuracy on readable outputs barely moves.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 35 references that are in the map. First posted on arXiv 11 Aug 2026.
Builds on
30 of 35 references.
References · 30
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2025Qwen3 Technical Reportcited by 8,820
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2025Gemma 3 Technical Reportcited by 1,824
- 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
- 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
- 2023GAIA: a benchmark for General AI Assistantscited by 1,128
18 more
- 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
- 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
- 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
- 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
- 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
- 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
- 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
- 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
- 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
- 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
- 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
- 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
- 2022TALM: Tool Augmented Language Modelscited by 197
- 2024How do Large Language Models Handle Multilingualism?cited by 150
- 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 136
- 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
- 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 118
Built on by
Citing papers per year: 2026: 1.
1 paper.