mlsanityData to 14 Oct 2026

Lumen-bench: Cross-Lingual Safety Gaps and the Say-Do Problem in Tool-Calling Large Language Models

arXiv 2610.15134 ↗First posted 10 Oct 2026cs.CLcited by 0

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

When a large language model refuses an unethical request, what does refusal mean? In conversation, refusal is bounded by what the model says; once the model is equipped with executable tools, what it does can diverge from what it says, a failure mode that English-only evaluations cannot fully reveal. We introduce Lumen-bench, a behavioral benchmark of tool-calling ethics spanning 30 models, five languages (Portuguese, English, Spanish, Mandarin Chinese, Hindi), 108 Portuguese-authored scenarios, six social-pressure conditions, two system prompts (ethical and neutral), and 21 generic tools. The evaluation comprises 194,400 trials. Judgments come from a reasoning-model judge (o4-mini), calibrated against one Portuguese-speaking author (n=83; harmful/non-harmful kappa=0.78), compared with two other provider judges (Fleiss' kappa=0.78, n=607), and complemented by retrospective sandbox replay. Four findings emerge. (i) Within-model comparisons support higher harmful-action rates in Hindi and Portuguese than English under the ethical prompt, and Hindi and Spanish under the neutral prompt, after multiple-testing correction. (ii) Premium models have lower median violation rates, although price does not guarantee safety or competence. (iii) Replacing the ethical system prompt with a neutral one raises pooled violation rates from 2.76% to 13.64%. (iv) In 8.57% of substantive-call cases, refusal or ethical objection accompanies a harmful tool call: the judges consistently identify the say-do gap, although they differ in how many responses they assign to this category. Matched benign controls demonstrate legitimate task completion in 82.8% of 3,300 trials. Code and reproduction scripts are available at https://github.com/TropicAI-Research/lumen.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 22 of its 42 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

22 of 42 references.

References · 22

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  6. 2025OpenAI GPT-5 System Cardcited by 989
  7. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  8. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  9. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  10. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  11. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 306
  12. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
10 more
  1. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  2. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 75
  3. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
  4. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 65
  5. 2024PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Modelscited by 20
  6. 2026ClawSafety: "Safe" LLMs, Unsafe Agentscited by 17
  7. 2025LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Modelscited by 16
  8. 2025When Refusals Fail: Unstable Safety Mechanisms in Long-Context LLM Agentscited by 15
  9. 2025MAPS: A Multilingual Benchmark for Agent Performance and Securitycited by 7
  10. 2026Sabi'a-4 Technical Reportcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Tool Mediation Alters Refusal Mechanisms in Large Language Models
  2. 2026SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models
  3. 2026Decided Upstream, Written Late: Locating and Pricing the Cross-Lingual Refusal Circuit of a Multilingual MoE
  4. 2026Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models
  5. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal
  6. 2024The Art of Saying No: Contextual Noncompliance in Language Models
  7. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models
  8. 2026Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation