# Toward Embedding-Based Psychometrics: Structural Modeling of Assessment-Item Semantics With Contextual Scores

arXiv: https://arxiv.org/abs/2609.31976 · HTML: https://mlsanity.com/paper/2609.31976/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 38 references that are in the map.
- First posted on arXiv 25 Sep 2026.

## Builds on (most cited first, 7 of 7)

- [MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms](https://mlsanity.com/paper/1905.13319/) (2019)
- [A Variational Inference method for Bayesian variable selection](https://mlsanity.com/paper/2211.11383/) (2022)
- [Documents Are People and Words Are Items: A Psychometric Approach to Textual Data with Contextual Embeddings](https://mlsanity.com/paper/2509.08920/) (2025)
- [From Text to Parameters: Predicting Item Parameters from Embedding Regularization with Reliability and Design Ceilings](https://mlsanity.com/paper/2607.07141/) (2026)
- [Recovering Latent Structures after Variational Bayesian Variable Selection: Fit Assessment and Factor-Number Selection in Partially Exploratory Factor Analysis](https://mlsanity.com/paper/2607.07159/) (2026)
- [Scalable Text-Embedding-informed Cognitive Diagnosis of Large Language Models](https://mlsanity.com/paper/2603.14676/) (2026)
- [When Is a General Factor Distinguishable? Non-Proportionality, Stable Structure, and the Bifactor Decision](https://mlsanity.com/paper/2608.10731/) (2026)

## Abstract (arXiv)

Contextual scores represent assessment items through their similarities to reference words in an external corpus. We examine the semantic structure of scores for 40 TIMSS mathematics scored units using a partially specified two-step factor procedure. A search across factor counts identifies a persistent seven-group structure under the featured construction. Subsequent comparisons consistently favor a general dimension alongside group associations, although individual group memberships remain sensitive to some specification choices. Item examples distinguish recurring, cross-domain, sensitive, and imposed associations. Simpler and unrestricted references clarify the contribution and limits of the anchored representation: it improves on a single factor but does not achieve the lowest working Bayesian information criterion (BIC). A separate response benchmark compares three initial Q constructions and their Hull-PVAF revisions under higher-order and saturated attribute distributions. Among these diagnostic models, BIC favors the official content framework and the Akaike information criterion (AIC) favors its direct four-factor augmentation, but a matched unidimensional two-parameter logistic model has lower AIC and BIC than all twelve conditions. These findings support a conditional semantic representation while limiting direct diagnostic interpretation. We discuss learned text-assisted response calibration as a prospective application requiring a larger calibrated item bank and independent evaluation.