mlsanityData to 7 Oct 2026

InCA: Rethinking In-Car Conversational System Assessment Leveraging Large Language Models

arXiv 2311.07469 ↗First posted 13 Nov 2023cs.CL cs.AIcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The assessment of advanced generative large language models (LLMs) poses a significant challenge, given their heightened complexity in recent developments. Furthermore, evaluating the performance of LLM-based applications in various industries, as indicated by Key Performance Indicators (KPIs), is a complex undertaking. This task necessitates a profound understanding of industry use cases and the anticipated system behavior. Within the context of the automotive industry, existing evaluation metrics prove inadequate for assessing in-car conversational question answering (ConvQA) systems. The unique demands of these systems, where answers may relate to driver or car safety and are confined within the car domain, highlight the limitations of current metrics. To address these challenges, this paper introduces a set of KPIs tailored for evaluating the performance of in-car ConvQA systems, along with datasets specifically designed for these KPIs. A preliminary and comprehensive empirical evaluation substantiates the efficacy of our proposed approach. Furthermore, we investigate the impact of employing varied personas in prompts and found that it enhances the model's capacity to simulate diverse viewpoints in assessments, mirroring how individuals with different backgrounds perceive a topic.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 8 of its 31 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

8 of 31 references.

References · 8

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  4. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  5. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  6. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  7. 2023Shepherd: A Critic for Language Model Generationcited by 96
  8. 2023Human Feedback is not Gold Standardcited by 82

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2025Automated Factual Benchmarking for In-Car Conversational Systems using Large Language Modelscited by 1
  2. 2024An Artificial Intelligence Approach for Interpreting Creative Combinational Designscited by 0
  3. 2026Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistantscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Benchmarking Contextual Understanding for In-Car Conversational Systems
  2. 2023CarExpert: Leveraging Large Language Models for In-Car Conversational Question Answering
  3. 2024Evaluating the Retrieval Component in LLM-Based Question Answering Systems
  4. 2025Do LLMs Understand Romanian Driving Laws? A Study on Multimodal and Fine-Tuned Question Answering
  5. 2025ChatGPT on the Road: Leveraging Large Language Model-Powered In-vehicle Conversational Agents for Safer and More Enjoyable Driving Experience
  6. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
  7. 2024Rethinking Generative Large Language Model Evaluation for Semantic Comprehension
  8. 2026STELLAR: A Search-Based Testing Framework for Large Language Model Applications