# Gender inference: can chatGPT outperform common commercial tools?

arXiv: https://arxiv.org/abs/2312.00805 · HTML: https://mlsanity.com/paper/2312.00805/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 52 references that are in the map.
- First posted on arXiv 24 Nov 2023.

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishing](https://mlsanity.com/paper/2303.13367/) (2023)
- [Historical comparison of gender inequality in scientific careers across countries and disciplines](https://mlsanity.com/paper/1907.04103/) (2019)
- [The extent and drivers of gender imbalance in neuroscience reference lists](https://mlsanity.com/paper/2001.01002/) (2020)
- [What's in a Name? -- Gender Classification of Names with Character Based Machine Learning Models](https://mlsanity.com/paper/2102.03692/) (2021)
- [Naive-Deep Face Recognition: Touching the Limit of LFW Benchmark or Not?](https://mlsanity.com/paper/1501.04690/) (2015)
- [Quantitative Evaluation of Gender Bias in Astronomical Publications from Citation Counts](https://mlsanity.com/paper/1610.08984/) (2016)
- [Inferring Gender from Names on the Web: A Comparative Evaluation of Gender Detection Methods](https://mlsanity.com/paper/1603.04322/) (2016)
- [Gender trends in computer science authorship](https://mlsanity.com/paper/1906.07883/) (2019)

## Built on by (most cited first, 2 of 2)

- [Beyond Binary Gender Labels: Revealing Gender Biases in LLMs through Gender-Neutral Name Predictions](https://mlsanity.com/paper/2407.05271/) (2024)
- [For the Misgendered Chinese in Gender Bias Research: Multi-Task Learning with Knowledge Distillation for Pinyin Name-Gender Prediction](https://mlsanity.com/paper/2405.06221/) (2024)

## Abstract (arXiv)

An increasing number of studies use gender information to understand phenomena such as gender bias, inequity in access and participation, or the impact of the Covid pandemic response. Unfortunately, most datasets do not include self-reported gender information, making it necessary for researchers to infer gender from other information, such as names or names and country information. An important limitation of these tools is that they fail to appropriately capture the fact that gender exists on a non-binary scale, however, it remains important to evaluate and compare how well these tools perform in a variety of contexts. In this paper, we compare the performance of a generative Artificial Intelligence (AI) tool ChatGPT with three commercially available list-based and machine learning-based gender inference tools (Namsor, Gender-API, and genderize.io) on a unique dataset. Specifically, we use a large Olympic athlete dataset and report how variations in the input (e.g., first name and first and last name, with and without country information) impact the accuracy of their predictions. We report results for the full set, as well as for the subsets: medal versus non-medal winners, athletes from the largest English-speaking countries, and athletes from East Asia. On these sets, we find that Namsor is the best traditional commercially available tool. However, ChatGPT performs at least as well as Namsor and often outperforms it, especially for the female sample when country and/or last name information is available. All tools perform better on medalists versus non-medalists and on names from English-speaking countries. Although not designed for this purpose, ChatGPT may be a cost-effective tool for gender prediction. In the future, it might even be possible for ChatGPT or other large scale language models to better identify self-reported gender rather than report gender on a binary scale.