Unsupervised Evaluation of Multi-Turn Objective-Driven Interactions
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) have seen increasing popularity in enterprise applications where AI agents and humans engage in objective-driven interactions. However, these systems are difficult to evaluate: data may be complex and unlabeled; human annotation is often impractical at scale; custom metrics can monitor for specific errors, but not previously-undetected ones; and LLM judges can produce unreliable results. We introduce the first set of unsupervised metrics for objective-driven interactions, leveraging statistical properties of unlabeled interaction data and using fine-tuned LLMs to adapt to distributional shifts. We develop metrics for labeling user goals, measuring goal completion, and quantifying LLM uncertainty without grounding evaluations in human-generated ideal responses. Our approach is validated on open-domain and task-specific interaction data.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 4 Nov 2025.
Builds on
30 of 43 references.
References · 30
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2017Decoupled Weight Decay Regularizationcited by 1,722
- 2024A Survey on LLM-as-a-Judgecited by 1,384
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
- 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
- 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
- 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
- 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
18 more
- 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
- 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
- 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
- 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
- 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
- 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
- 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Modelscited by 97
- 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 89
- 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 73
- 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
- 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
- 2025Specializing Large Language Models to Simulate Survey Response Distributions for Global Populationscited by 50
- 2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judgecited by 33
- 2025Validating LLM-as-a-Judge Systems under Rating Indeterminacycited by 23
- 2021BLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-oriented Textcited by 22
- 2025HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Taskscited by 16
- 2025UQLM: A Python Package for Uncertainty Quantification in Large Language Modelscited by 9
- 2024Rethinking Scale: The Efficacy of Fine-Tuned Open-Source LLMs in Large-Scale Reproducible Social Science Researchcited by 6
Built on by
Not cited yet.