mlsanityData to 7 Oct 2026

A Japanese Benchmark for Evaluating Social Bias in Reasoning Based on Attribution Theory

arXiv 2604.00568 ↗First posted 1 Apr 2026cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In enhancing the fairness of Large Language Models (LLMs), evaluating social biases rooted in the cultural contexts of specific linguistic regions is essential. However, most existing Japanese benchmarks heavily rely on translating English data, which does not necessarily provide an evaluation suitable for Japanese culture. Furthermore, they only evaluate bias in the conclusion, failing to capture biases lurking in the reasoning. In this study, based on attribution theory in social psychology, we constructed a new dataset, ``JUBAKU-v2,'' which evaluates the bias in attributing behaviors to in-groups and out-groups within reasoning while fixing the conclusion. This dataset consists of 216 examples reflecting cultural biases specific to Japan. Experimental results verified that it can detect performance differences across models more sensitively than existing benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 6 of its 20 references that are in the map. First posted on arXiv 1 Apr 2026.

Builds on

6 of 20 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2024Evaluating Gender Bias in Large Language Models via Chain-of-Thought Prompting28 Jan

Other references · 4

  1. 2024GPT-4o System Cardcited by 4,467
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  3. 2025Intent-Aware Self-Correction for Mitigating Social Biases in Large Language Modelscited by 5
  4. 2026JUBAKU: An Adversarial Benchmark for Exposing Culturally Grounded Stereotypes in Japanese LLMscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024JBBQ: Japanese Bias Benchmark for Analyzing Social Biases in Large Language Models
  2. 2025BiasCause: Evaluate Socially Biased Causal Reasoning of Large Language Models
  3. 2025Investigating Thinking Behaviours of Reasoning-Based Language Models for Social Bias Mitigation
  4. 2025Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition
  5. 2026Debiasing Large Language Models toward Social Factors in Online Behavior Analytics through Prompt Knowledge Tuning
  6. 2025Does Reasoning Introduce Bias? A Study of Social Bias Evaluation and Mitigation in LLM Reasoning
  7. 2026CuBEs: Culturally-Situated Behavioral Evaluations and the Limitations of Culture-Blind LLM Judges
  8. 2025Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection