mlsanityData to 14 Oct 2026

Proxy-Validated LLM UX Micro-Simulations: An Artifact-First Protocol for Early-Stage Decision Support

arXiv 2608.13563 ↗First posted 3 Jun 2026cs.HC cs.AI cs.SEcited by 0

Not on the map. Why?

Abstract (arXiv)

Early-stage teams often lack users, time, and budget to run repeated UX studies, yet still need decision-oriented signals to iterate safely. We study an LLM-driven UX micro-simulation pipeline that generates structured customer-experience feedback (walkthrough steps, friction points, micro-survey signals) from versioned prompts, personas, tasks, and UI snapshots. Because public usability datasets with task outcomes are scarce, we validate simulated friction themes using multiple public proxy corpora (app reviews, support tweets, and open-source software issues). We propose a lightweight proxy-validation protocol with two alignment metrics: top-k Jaccard and distributional weighted-Jaccard (W), and compare lexical, TF-IDF, and multilingual embedding baselines across six proxy datasets. Embedding-based alignment yields higher W than lexical baselines on primary app-review and support-tweet proxies (e.g., W=0.128 vs 0.000 on Gojek), while top-k Jaccard is shown to overstate alignment at large k. We ablate four agent strategies (single-pass, best-of-N, hybrid, and a proposed score-then-select judge) across Azure OpenAI deployments and report bootstrap confidence intervals over 8 method-dataset pairs; these intervals reveal that the embedding W point estimate is systematically unstable under resampling at our subsample size. We also provide a failure-mode analysis of grounding and fabrication proxies, with documented calibration caveats and worked examples of outputs flagged as fabricated by an adversarial judge. Our artifact-first pipeline produces reproducible tables and figures from versioned run artifacts, supporting iterative prompt and taxonomy refinement before final paid-model calibration.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 26 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

13 of 26 references.

References · 13

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  5. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  6. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
  7. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  8. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  9. 2024AgentReview: Exploring Peer Review Dynamics with LLM Agentscited by 93
  10. 2025LLM Generated Persona is a Promise with a Catchcited by 81
  11. 2024Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machinacited by 29
  12. 2024Proxona: Supporting Creators' Sensemaking and Ideation with LLM-Powered Audience Personascited by 19
1 more
  1. 2023Framework-Based Qualitative Analysis of Free Responses of Large Language Models: Algorithmic Fidelitycited by 7

Built on by

Not cited yet.