mlsanityData to 14 Oct 2026

Predicting Question-Answering Performance of Large Language Models through Semantic Consistency

arXiv 2311.01152 ↗First posted 2 Nov 2023cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

Semantic consistency of a language model is broadly defined as the model's ability to produce semantically-equivalent outputs, given semantically-equivalent inputs. We address the task of assessing question-answering (QA) semantic consistency of contemporary large language models (LLMs) by manually creating a benchmark dataset with high-quality paraphrases for factual questions, and release the dataset to the community. We further combine the semantic consistency metric with additional measurements suggested in prior work as correlating with LLM QA accuracy, for building and evaluating a framework for factual QA reference-less performance prediction -- predicting the likelihood of a language model to accurately answer a question. Evaluating the framework on five contemporary LLMs, we demonstrate encouraging, significantly outperforming baselines, results.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 12 of its 28 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

12 of 28 references.

References · 12

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  7. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  8. 2022Prompt Consistency for Zero-Shot Task Generalizationcited by 67
  9. 2022Measuring Reliability of Large Language Models through Semantic Consistencycited by 38
  10. 2022Quality Controlled Paraphrase Generationcited by 31
  11. 2021P-Adapters: Robustly Extracting Factual Information from Language Models with Diverse Promptscited by 22
  12. 2021Accurate, yet inconsistent? Consistency Analysis on Language Understanding Modelscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 13; 2026: 4.

28 papers.

Sort:

By area:

  1. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  2. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 50
  3. 2024Harnessing the Power of Multiple Minds: Lessons Learned from LLM Routingcited by 46
  4. 2025Consistently Simulating Human Personas with Multi-Turn Reinforcement Learningcited by 39
  5. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Modelscited by 28
  6. 2025On the Robustness of Agentic Function Callingcited by 19
  7. 2024From Imitation to Introspection: Probing Self-Consciousness in Language Modelscited by 14
  8. 2025Enhancing Semantic Consistency of Large Language Models through Model Editing: An Interpretability-Oriented Approachcited by 14
  9. 2025Consistency in Language Models: Current Landscape, Challenges, and Future Directionscited by 14
  10. 2025Improving Consistency in Large Language Models through Chain of Guidancecited by 13
  11. 2024A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarioscited by 8
  12. 2024SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Modelscited by 8
16 more
  1. 2025LF-Steering: Latent Feature Activation Steering for Enhancing Semantic Consistency in Large Language Modelscited by 8
  2. 2025Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Modelscited by 5
  3. 2024Is Cognition Consistent with Perception? Assessing and Mitigating Multimodal Knowledge Conflicts in Document Understandingcited by 4
  4. 2025Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Modelscited by 4
  5. 2025Cleanse: Uncertainty Estimation Approach Using Clustering-based Semantic Consistency in LLMscited by 3
  6. 2026Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliabilitycited by 3
  7. 2024A Nurse is Blue and Elephant is Rugby: Cross Domain Alignment in Large Language Models Reveal Human-like Patternscited by 2
  8. 2024Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responsescited by 2
  9. 2025Anticipatory Evaluation of Language Models: Predicting LLM Benchmark Scores from Descriptionscited by 2
  10. 2025Improving Consistency in Retrieval-Augmented Systems with Group Similarity Rewardscited by 2
  11. 2026GSM-SEM: Benchmark and Framework for Generating Semantically Variant Augmentationscited by 2
  12. 2024Using Combinatorial Optimization to Design a High quality LLM Solutioncited by 0
  13. 2025Improving Document Retrieval Coherence for Semantically Equivalent Queriescited by 0
  14. 2025Entropy-Constrained Strategy Optimization in Urban Floods: A Multi-Agent Framework with LLM and Knowledge Graph Integrationcited by 0
  15. 2026A Reproducibility Study of Metacognitive Retrieval-Augmented Generationcited by 0
  16. 2026ASTRA-QA: A Benchmark for Abstract Question Answering over Documentscited by 0