mlsanityData to 14 Oct 2026

Amulet: Putting Complex Multi-Turn Conversations on the Stand with LLM Juries

arXiv 2505.20451 ↗First posted 26 May 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Today, large language models are widely used as judges to evaluate responses from other language models. Hence, it is imperative to benchmark and improve these LLM-judges on real-world language model usage: a typical human-assistant conversation is lengthy, and shows significant diversity in topics, intents, and requirements across turns, e.g. social interactions, task requests, feedback. We present Amulet, a framework that leverages pertinent linguistic concepts of dialog-acts and maxims to improve the accuracy of LLM-judges on preference data with complex, multi-turn conversational context. Amulet presents valuable insights about (a) the communicative structures and intents present in the conversation (dialog acts), and (b) the satisfaction of conversational principles (maxims) by the preference responses, and uses them to make judgments. On four challenging datasets, Amulet shows that (a) humans frequently (60 to 70 percent of the time) change their intents from one turn of the conversation to the next, and (b) in 75 percent of instances, the preference responses can be differentiated via dialog acts and/or maxims, reiterating the latter's significance in judging such data. Amulet can be used either as a judge by applying the framework to a single LLM, or integrated into a jury with different LLM judges; our judges and juries show strong improvements on relevant baselines for all four datasets.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 55 of its 94 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

55 of 94 references.

References · 55

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  9. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  10. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  11. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  12. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
38 more
  1. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  3. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  4. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  5. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  6. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  7. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  8. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  9. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  10. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  11. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 309
  12. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
  13. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  14. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  15. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  16. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
  17. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  18. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 131
  19. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  20. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 105
  21. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
  22. 2024Can LLM be a Personalized Judge?cited by 100
  23. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  24. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  25. 2023Alleviating Hallucinations of Large Language Models through Induced Hallucinationscited by 79
  26. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 74
  27. 2022In conversation with Artificial Intelligence: aligning language models with human valuescited by 62
  28. 2024Quantile Regression for Distributional Reward Models in RLHFcited by 60
  29. 2020A Taxonomy of Empathetic Response Intents in Human Social Conversationscited by 51
  30. 2022Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversalcited by 40
  31. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  32. 2024Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinationscited by 38
  33. 2019Dialogue Act Classification with Context-Aware Self-Attentioncited by 26
  34. 2020Guiding attention in Sequence-to-sequence models for Dialogue Act predictioncited by 26
  35. 2024Language Models in Dialogue: Conversational Maxims for Human-AI Interactionscited by 21
  36. 2024Aligning Large Language Models by On-Policy Self-Judgmentcited by 20
  37. 2019User Intent Prediction in Information-seeking Conversationscited by 19
  38. 2017Dialogue Act Recognition via CRF-Attentive Structured Networkcited by 18

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.