mlsanityData to 14 Oct 2026

Effectiveness of Zero-shot-CoT in Japanese Prompts

arXiv 2503.06765 ↗First posted 9 Mar 2025cs.CL cs.AIcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

We compare the effectiveness of zero-shot Chain-of-Thought (CoT) prompting in Japanese and English using ChatGPT-3.5 and 4o-mini. The technique of zero-shot CoT, which involves appending a phrase such as "Let's think step by step" to a prompt to encourage reasoning before answering, has been shown to offer LLM performance improvements in mathematical and reasoning tasks, particularly in English. We investigate how these effects transfer to Japanese using the Japanese Multi-task Language Understanding Benchmark (JMMLU) and the Multi-task Language Understanding Benchmark (MMLU). Our results show that while zero-shot CoT prompting can lead to notable performance gains for some prompt categories in GPT-3.5, its impact in GPT-4o-mini is associated with significant performance declines. However, for Japanese prompts there remain certain categories, such as college mathematics and abstract algebra, that still exhibit improvements, despite the broader trend of diminishing effectiveness in more advanced models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 9 of its 12 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

9 of 12 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024The Impact of Reasoning Step Length on Large Language Models10 Jan
  2. 2022Large Language Models are Zero-Shot Reasoners24 May
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 6

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  5. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 146
  6. 2024Should We Respect LLMs? A Cross-Lingual Study on the Influence of Prompt Politeness on LLM Performancecited by 41

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve
  2. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages
  3. 2025Revisiting Chain-of-Thought Prompting: Zero-shot Can Be Stronger than Few-shot
  4. 2023Stress Testing Chain-of-Thought Prompting for Large Language Models
  5. 2024Instance-adaptive Zero-shot Chain-of-Thought Prompting
  6. 2023Towards Better Chain-of-Thought Prompting Strategies: A Survey
  7. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
  8. 2023An automatically discovered chain-of-thought prompt generalizes to novel models and datasets