# Using LLMs to identify features of personal and professional skills in an open-response situational judgment test

arXiv: https://arxiv.org/abs/2507.13881 · HTML: https://mlsanity.com/paper/2507.13881/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 1 of its 27 references that are in the map.
- First posted on arXiv 18 Jul 2025.

## Builds on (most cited first, 1 of 1)

- [Applying Large Language Models and Chain-of-Thought for Automatic Scoring](https://mlsanity.com/paper/2312.03748/) (2023)

## Built on by (most cited first, 3 of 3)

- [Measuring What Matters -- or What's Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factors](https://mlsanity.com/paper/2603.25674/) (2026)
- [Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests](https://mlsanity.com/paper/2510.22170/) (2025)
- [ADE: Agentic Data Evolution Framework for Human-Centered Objectives](https://mlsanity.com/paper/2608.23719/) (2026)

## Abstract (arXiv)

Academic programs are increasingly recognizing the importance of personal and professional skills and their critical role alongside technical expertise in preparing students for future success in diverse career paths. With this growing demand comes the need for scalable systems to measure, evaluate, and develop these skills. Situational Judgment Tests (SJTs) offer one potential avenue for measuring these skills in a standardized and reliable way, but open-response SJTs have traditionally relied on trained human raters for evaluation, presenting operational challenges to delivering SJTs at scale. Past attempts at developing NLP-based scoring systems for SJTs have fallen short due to issues with construct validity of these systems. In this article, we explore a novel approach to extracting construct-relevant features from SJT responses using large language models (LLMs). We use the Casper SJT to demonstrate the efficacy of this approach. This study sets the foundation for future developments in automated scoring for personal and professional skills.