# Assessing Personalized AI Mentoring with Large Language Models in the Computing Field

arXiv: https://arxiv.org/abs/2412.08430 · HTML: https://mlsanity.com/paper/2412.08430/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 9 of 9)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Large Language Models for Education: A Survey](https://mlsanity.com/paper/2405.13001/) (2024)
- [Large Language Models (GPT) for automating feedback on programming assignments](https://mlsanity.com/paper/2307.00150/) (2023)
- [Exploring the Role of AI Assistants in Computer Science Education: Methods, Implications, and Instructor Perspectives](https://mlsanity.com/paper/2306.03289/) (2023)
- [AI-Tutoring in Software Engineering Education](https://mlsanity.com/paper/2404.02548/) (2024)
- [Zero-shot Learning with Minimum Instruction to Extract Social Determinants and Family History from Clinical Notes using GPT Model](https://mlsanity.com/paper/2309.05475/) (2023)
- [Designing an AI-Powered Mentorship Platform for Professional Development: Opportunities and Challenges](https://mlsanity.com/paper/2407.20233/) (2024)

## Built on by (most cited first, 2 of 2)

- [New Kid in the Classroom: Exploring Student Perceptions of AI Coding Assistants](https://mlsanity.com/paper/2507.22900/) (2025)
- [Adaptive Learning Systems: Personalized Curriculum Design Using LLM-Powered Analytics](https://mlsanity.com/paper/2507.18949/) (2025)

## Abstract (arXiv)

This paper provides an in-depth evaluation of three state-of-the-art Large Language Models (LLMs) for personalized career mentoring in the computing field, using three distinct student profiles that consider gender, race, and professional levels. We evaluated the performance of GPT-4, LLaMA 3, and Palm 2 using a zero-shot learning approach without human intervention. A quantitative evaluation was conducted through a custom natural language processing analytics pipeline to highlight the uniqueness of the responses and to identify words reflecting each student's profile, including race, gender, or professional level. The analysis of frequently used words in the responses indicates that GPT-4 offers more personalized mentoring compared to the other two LLMs. Additionally, a qualitative evaluation was performed to see if human experts reached similar conclusions. The analysis of survey responses shows that GPT-4 outperformed the other two LLMs in delivering more accurate and useful mentoring while addressing specific challenges with encouragement languages. Our work establishes a foundation for developing personalized mentoring tools based on LLMs, incorporating human mentors in the process to deliver a more impactful and tailored mentoring experience.