# LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation

arXiv: https://arxiv.org/abs/2412.10424 · HTML: https://mlsanity.com/paper/2412.10424/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Builds on (most cited first, 23 of 23)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [OLMoE: Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2409.02060/) (2024)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues](https://mlsanity.com/paper/2402.14762/) (2024)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [Don't Make Your LLM an Evaluation Benchmark Cheater](https://mlsanity.com/paper/2311.01964/) (2023)
- [Investigating Data Contamination for Pre-training Language Models](https://mlsanity.com/paper/2401.06059/) (2024)
- [ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisons](https://mlsanity.com/paper/1909.03087/) (2019)
- [Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation](https://mlsanity.com/paper/2402.11443/) (2024)
- [KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models](https://mlsanity.com/paper/2402.15043/) (2024)
- [Evading Data Contamination Detection for Language Models is (too) Easy](https://mlsanity.com/paper/2402.02823/) (2024)
- [Hierarchical Deconstruction of LLM Reasoning: A Graph-Based Framework for Analyzing Knowledge Utilization](https://mlsanity.com/paper/2406.19502/) (2024)
- [LogProber: Disentangling confidence from contamination in LLM responses](https://mlsanity.com/paper/2408.14352/) (2024)

## Built on by (most cited first, 13 of 13)

- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation](https://mlsanity.com/paper/2502.17521/) (2025)
- [RefineBench: Evaluating Refinement Capability of Language Models via Checklists](https://mlsanity.com/paper/2511.22173/) (2025)
- [Flex-TravelPlanner: A Benchmark for Flexible Planning with Language Agents](https://mlsanity.com/paper/2506.04649/) (2025)
- [Cross-Context Review: Improving LLM Output Quality by Separating Production and Review Sessions](https://mlsanity.com/paper/2603.12123/) (2026)
- [More Rounds, More Noise: Why Multi-Turn Review Fails to Improve Cross-Context Verification](https://mlsanity.com/paper/2603.16244/) (2026)
- [Checklist Engineering Empowers Multilingual LLM Judges](https://mlsanity.com/paper/2507.06774/) (2025)
- [JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation](https://mlsanity.com/paper/2509.02097/) (2025)
- [Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation](https://mlsanity.com/paper/2510.13884/) (2025)
- [Automated Safety Benchmarking: A Multi-agent Pipeline for LVLMs](https://mlsanity.com/paper/2601.19507/) (2026)
- [YIELD: A Large-Scale Dataset and Evaluation Framework for Information Elicitation Agents](https://mlsanity.com/paper/2604.10968/) (2026)
- [DyMT-ESB: Dynamic Multi-Turn Evaluation of Social Bias in User-LLM Interactions](https://mlsanity.com/paper/2609.18649/) (2026)
- [On Open-Ended Information Seeking for Information Elicitation Agents](https://mlsanity.com/paper/2610.07509/) (2026)

## Abstract (arXiv)

We introduce LLM-as-an-Interviewer, a novel paradigm for evaluating large language models (LLMs). This approach leverages multi-turn interactions where the LLM interviewer actively provides feedback on responses and poses follow-up questions to the evaluated LLM. At the start of the interview, the LLM interviewer dynamically modifies datasets to generate initial questions, mitigating data contamination. We apply the LLM-as-an-Interviewer framework to evaluate six models on the MATH and DepthQA tasks. Our results show that the framework effectively provides insights into LLM performance, including the quality of initial responses, adaptability to feedback, and ability to address follow-up queries like clarification or additional knowledge requests. The framework also addresses key limitations of conventional methods like LLM-as-a-Judge, including verbosity bias and inconsistency across runs. Finally, we propose the Interview Report, which aggregates insights from the interview process, providing examples and a comprehensive analysis of the LLM's strengths and weaknesses. This report offers a detailed snapshot of the model's real-world applicability. The code for our framework is publicly available at https://github.com/interview-eval/.