mlsanityData to 7 Oct 2026

The power of Prompts: Evaluating and Mitigating Gender Bias in MT with LLMs

arXiv 2407.18786 ↗First posted 26 Jul 2024cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper studies gender bias in machine translation through the lens of Large Language Models (LLMs). Four widely-used test sets are employed to benchmark various base LLMs, comparing their translation quality and gender bias against state-of-the-art Neural Machine Translation (NMT) models for English to Catalan (En rightarrow Ca) and English to Spanish (En rightarrow Es) translation directions. Our findings reveal pervasive gender bias across all models, with base LLMs exhibiting a higher degree of bias compared to NMT models. To combat this bias, we explore prompting engineering techniques applied to an instruction-tuned LLM. We identify a prompt structure that significantly reduces gender bias by up to 12% on the WinoMT evaluation dataset compared to more straightforward prompts. These results significantly reduce the gender bias accuracy gap between LLMs and traditional NMT systems.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 39 of its 68 references that are in the map. First posted on arXiv 26 Jul 2024.

Builds on

39 of 68 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  10. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  11. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  12. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
27 more
  1. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  2. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  3. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  4. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  5. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  6. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  7. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  8. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  9. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  10. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  11. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  12. 2019Evaluating Gender Bias in Machine Translationcited by 247
  13. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  14. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  15. 2022In-context Examples Selection for Machine Translationcited by 172
  16. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  17. 2021Gender Bias in Machine Translationcited by 116
  18. 2023Text Classification via Large Language Modelscited by 104
  19. 2023The unreasonable effectiveness of few-shot learning for machine translationcited by 91
  20. 2020Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problemcited by 86
  21. 2023Exploring the Feasibility of ChatGPT for Event Extractioncited by 76
  22. 2021Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translationcited by 62
  23. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 58
  24. 2020Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpuscited by 57
  25. 2020Mitigating Gender Bias in Machine Translation with Target Gender Annotationscited by 42
  26. 2024A Prompt Response to the Demand for Automatic Gender-Neutral Translationcited by 12
  27. 2020Gender Bias in Multilingual Neural Machine Translation: The Architecture Matterscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 4.

12 papers.

  1. 2024Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimationcited by 10
  2. 2025An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluationcited by 2
  3. 2025Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domainscited by 1
  4. 2024MT-LENS: An all-in-one Toolkit for Better Machine Translation Evaluationcited by 0
  5. 2025Auto-Search and Refinement: An Automated Framework for Gender Bias Mitigation in Large Language Modelscited by 0
  6. 2025Discrimination by LLMs: Cross-lingual Bias Assessment and Mitigation in Decision-Making and Summarisationcited by 0
  7. 2025Conveying Imagistic Thinking in Traditional Chinese Medicine Translation: A Prompt Engineering and LLM-Based Evaluation Frameworkcited by 0
  8. 2025Conveying Imagistic Thinking in Traditional Chinese Medicine Translation: A Prompt Engineering and LLM-Based Evaluation Frameworkcited by 0
  9. 2026MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generationcited by 0
  10. 2026Cultural Fidelity in English-to-Hindi Translation: A Preservation-Fluency Frontier for Gender Recoverabilitycited by 0
  11. 2026Contaminated Collaboration: Measuring Gender Bias Transfer in LLM-Assisted Student Writingcited by 0
  12. 2026Mitigating Gender Bias in English to Romanian Machine Translationcited by 0