mlsanityData to 14 Oct 2026

MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systems

arXiv 2412.15557 ↗First posted 20 Dec 2024cs.SE cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

With the widespread application of LLM-based dialogue systems in daily life, quality assurance has become more important than ever. Recent research has successfully introduced methods to identify unexpected behaviour in single-turn testing scenarios. However, multi-turn interaction is the common real-world usage of dialogue systems, yet testing methods for such interactions remain underexplored. This is largely due to the oracle problem in multi-turn testing, which continues to pose a significant challenge for dialogue system developers and researchers. In this paper, we propose MORTAR, a metamorphic multi-turn dialogue testing approach, which mitigates the test oracle problem in testing LLM-based dialogue systems. MORTAR formalises the multi-turn testing for dialogue systems, and automates the generation of question-answer dialogue test cases with multiple dialogue-level perturbations and metamorphic relations (MRs). The automated MR matching mechanism allows MORTAR more flexibility and efficiency in metamorphic testing. The proposed approach is fully automated without reliance on LLM judges. In testing six popular LLM-based dialogue systems, MORTAR reaches significantly better effectiveness with over 150% more bugs revealed per test case when compared to the single-turn metamorphic testing baseline. Regarding the quality of bugs, MORTAR reveals higher-quality bugs in terms of diversity, precision and uniqueness. MORTAR is expected to inspire more multi-turn testing approaches, and assist developers in evaluating the dialogue system performance more comprehensively with constrained test resources and budget.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 51 of its 87 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

51 of 87 references.

References · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  8. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2024Qwen2 Technical Reportcited by 2,120
  12. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
38 more
  1. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  2. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  4. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  5. 2018CoQA: A Conversational Question Answering Challengecited by 683
  6. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  7. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  8. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  9. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 504
  10. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  11. 2023ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPTcited by 250
  12. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  13. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 236
  14. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 218
  15. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  16. 2018Neural Approaches to Conversational AIcited by 165
  17. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  18. 2019Survey on Evaluation Methods for Dialogue Systemscited by 126
  19. 2020MuTual: A Dataset for Multi-Turn Dialogue Reasoningcited by 115
  20. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  21. 2020DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialoguecited by 89
  22. 2024MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Modelscited by 81
  23. 2020Metamorphic Testing: A New Approach for Generating Next Test Casescited by 75
  24. 2024Extract, Define, Canonicalize: An LLM-based Framework for Knowledge Graph Constructioncited by 70
  25. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  26. 2023BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialoguescited by 34
  27. 2018Anaphora and Coreference Resolution: A Reviewcited by 31
  28. 2023DialogBench: Evaluating LLMs as Human-like Dialogue Systemscited by 31
  29. 2023MTTM: Metamorphic Testing for Textual Content Moderation Softwarecited by 22
  30. 2024A Survey of Conversational Searchcited by 22
  31. 2023IterCQR: Iterative Conversational Query Reformulation with Retrieval Guidancecited by 20
  32. 2023METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualitiescited by 20
  33. 2023Parrot: Enhancing Multi-Turn Instruction Following for Large Language Modelscited by 16
  34. 2024Drowzee: Metamorphic Testing for Fact-Conflicting Hallucination Detection in Large Language Modelscited by 14
  35. 2019Filling Conversation Ellipsis for Better Social Dialog Understandingcited by 7
  36. 2023Software Testing of Generative AI Systems: Challenges and Opportunitiescited by 7
  37. 2024As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Losscited by 6
  38. 2022OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialoguecited by 4

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 1.

7 papers.

  1. 2025Metamorphic Testing of Large Language Models for Natural Language Processingcited by 12
  2. 2025Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocolcited by 6
  3. 2025Automated Validation of LLM-based Evaluators for Software Engineering Artifactscited by 4
  4. 2025MetaRAG: Metamorphic Testing for Hallucination Detection in RAG Systemscited by 2
  5. 2025HFuzzer: Testing Large Language Models for Package Hallucinations via Phrase-based Fuzzingcited by 1
  6. 2025LLM Assisted Coding with Metamorphic Specification Mutation Agentcited by 1
  7. 2026Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistantscited by 0