# NC-Bench: An LLM Benchmark for Evaluating Conversational Competence

arXiv: https://arxiv.org/abs/2601.06426 · HTML: https://mlsanity.com/paper/2601.06426/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 10 Jan 2026.

## Builds on (most cited first, 21 of 21)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset](https://mlsanity.com/paper/1710.03957/) (2017)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [Benchmarking Large Language Models in Retrieval-Augmented Generation](https://mlsanity.com/paper/2309.01431/) (2023)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs](https://mlsanity.com/paper/2501.17399/) (2025)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [A Complete Survey on LLM-based AI Chatbots](https://mlsanity.com/paper/2406.16937/) (2024)
- [Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey](https://mlsanity.com/paper/2503.22458/) (2025)
- [MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2501.03468/) (2025)
- [A Comprehensive Analysis of the Effectiveness of Large Language Models as Automatic Dialogue Evaluators](https://mlsanity.com/paper/2312.15407/) (2023)
- [LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routing](https://mlsanity.com/paper/2502.09977/) (2025)
- [A Survey of the Evolution of Language Model-Based Dialogue Systems: Data, Task and Models](https://mlsanity.com/paper/2311.16789/) (2023)
- [Evaluating LLM Metrics Through Real-World Capabilities](https://mlsanity.com/paper/2505.08253/) (2025)
- [Do Large Language Models Need Intent? Revisiting Response Generation Strategies for Service Assistant](https://mlsanity.com/paper/2509.05006/) (2025)
- [Zero-shot Conversational Summarization Evaluations with small Large Language Models](https://mlsanity.com/paper/2311.18041/) (2023)
- [Does Model Size Matter? A Comparison of Small and Large Language Models for Requirements Classification](https://mlsanity.com/paper/2510.21443/) (2025)

## Built on by (most cited first, 1 of 1)

- [Limits of Reliability and Scaling in Language Models](https://mlsanity.com/paper/2607.14112/) (2026)

## Abstract (arXiv)

Existing LLM benchmarks evaluate what models say, such as whether answers are correct, faithful, or helpful, but they do not test whether models produce the right type of conversational action at the right point in an interaction. The Natural Conversation Benchmark (NC-Bench) fills this gap by evaluating conversational competence: the ability to perform structurally appropriate actions such as repairing, closing, or refusing, as defined by conversation science. Grounded in the Natural Conversation Framework (NCF), NC-Bench comprises three sets: (1) the basic set evaluates fundamental sequence management practices, such as answering inquiries, repairing responses, and closing conversational pairs; (2) the retrieval-augmented generation (RAG) set applies the same patterns but incorporates information-seeking via RAG; (3) the complex request set extends to requests involving more intricate sequence management. Each set tests a model's ability to produce contextually appropriate conversational actions in response to characteristic interaction patterns. Evaluations across six open-source models and one closed-source model on 14 interaction patterns reveal quantifiable shortcomings in conversational competence present in current models. By operationalizing fundamental principles of human conversation, NC-Bench provides a lightweight, extensible, and theory-grounded framework for identifying specific conversational action gaps in LLMs beyond topical or task-specific benchmarks.