mlsanityData to 14 Oct 2026

Investigating Bias: A Multilingual Pipeline for Generating, Solving, and Evaluating Math Problems with LLMs

arXiv 2509.17701 ↗First posted 22 Sep 2025cs.CL cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are increasingly used for educational support, yet their response quality varies depending on the language of interaction. This paper presents an automated multilingual pipeline for generating, solving, and evaluating math problems aligned with the German K-10 curriculum. We generated 628 math exercises and translated them into English, German, and Arabic. Three commercial LLMs (GPT-4o-mini, Gemini 2.5 Flash, and Qwen-plus) were prompted to produce step-by-step solutions in each language. A held-out panel of LLM judges, including Claude 3.5 Haiku, evaluated solution quality using a comparative framework. Results show a consistent gap, with English solutions consistently rated highest, and Arabic often ranked lower. These findings highlight persistent linguistic bias and the need for more equitable multilingual AI systems in education.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 13 of its 18 references that are in the map. First posted on arXiv 22 Sep 2025.

Builds on

13 of 18 references.

References · 13

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  3. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  4. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  5. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  6. 2023Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Reviewcited by 83
  7. 2025Do Multilingual LLMs Think In English?cited by 78
  8. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 71
  9. 2023Large Language Models in Education: Vision and Opportunitiescited by 54
  10. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 43
  11. 2023Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challengescited by 13
  12. 2025Multilingual Performance Biases of Large Language Models in Educationcited by 9
1 more
  1. 2024A Course Intelligence Platform for Higher Education: Lessons from AI-Assisted Course Evaluationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoningcited by 6
  2. 2026REAgent: Requirement-Driven LLM Agents for Software Issue Resolutioncited by 4
  3. 2025GPT and Prejudice: A Sparse Approach to Understanding Learned Representations in Large Language Modelscited by 0
  4. 2026A self-evolving multi-role collaborative framework with fine-grained difficulty guidance for innovative mathematical problem generationcited by 0