# Investigating Bias: A Multilingual Pipeline for Generating, Solving, and Evaluating Math Problems with LLMs

arXiv: https://arxiv.org/abs/2509.17701 · HTML: https://mlsanity.com/paper/2509.17701/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 22 Sep 2025.

## Builds on (most cited first, 13 of 13)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [JudgeBench: A Benchmark for Evaluating LLM-based Judges](https://mlsanity.com/paper/2410.12784/) (2024)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators](https://mlsanity.com/paper/2403.16950/) (2024)
- [Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review](https://mlsanity.com/paper/2303.13379/) (2023)
- [Do Multilingual LLMs Think In English?](https://mlsanity.com/paper/2502.15603/) (2025)
- [LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models](https://mlsanity.com/paper/2307.07889/) (2023)
- [Large Language Models in Education: Vision and Opportunities](https://mlsanity.com/paper/2311.13160/) (2023)
- [Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap](https://mlsanity.com/paper/2501.02448/) (2025)
- [Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challenges](https://mlsanity.com/paper/2401.08664/) (2023)
- [Multilingual Performance Biases of Large Language Models in Education](https://mlsanity.com/paper/2504.17720/) (2025)
- [A Course Intelligence Platform for Higher Education: Lessons from AI-Assisted Course Evaluation](https://mlsanity.com/paper/2411.02455/) (2024)

## Built on by (most cited first, 4 of 4)

- [MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning](https://mlsanity.com/paper/2510.14305/) (2025)
- [REAgent: Requirement-Driven LLM Agents for Software Issue Resolution](https://mlsanity.com/paper/2604.06861/) (2026)
- [GPT and Prejudice: A Sparse Approach to Understanding Learned Representations in Large Language Models](https://mlsanity.com/paper/2510.01252/) (2025)
- [A self-evolving multi-role collaborative framework with fine-grained difficulty guidance for innovative mathematical problem generation](https://mlsanity.com/paper/2601.11792/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used for educational support, yet their response quality varies depending on the language of interaction. This paper presents an automated multilingual pipeline for generating, solving, and evaluating math problems aligned with the German K-10 curriculum. We generated 628 math exercises and translated them into English, German, and Arabic. Three commercial LLMs (GPT-4o-mini, Gemini 2.5 Flash, and Qwen-plus) were prompted to produce step-by-step solutions in each language. A held-out panel of LLM judges, including Claude 3.5 Haiku, evaluated solution quality using a comparative framework. Results show a consistent gap, with English solutions consistently rated highest, and Arabic often ranked lower. These findings highlight persistent linguistic bias and the need for more equitable multilingual AI systems in education.