mlsanityData to 13 Oct 2026

A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity

arXiv 2302.04023 ↗First posted 8 Feb 2023cs.CL cs.AIcited by 894

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

This paper proposes a framework for quantitatively evaluating interactive LLMs such as ChatGPT using publicly available data sets. We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. We find that ChatGPT outperforms LLMs with zero-shot learning on most tasks and even outperforms fine-tuned models on some tasks. We find that it is better at understanding non-Latin script languages than generating them. It is able to generate multimodal content from textual prompts, via an intermediate code generation step. Moreover, we find that ChatGPT is 63.41% accurate on average in 10 different reasoning categories under logical reasoning, non-textual reasoning, and commonsense reasoning, hence making it an unreliable reasoner. It is, for example, better at deductive than inductive reasoning. ChatGPT suffers from hallucination problems like other LLMs and it generates more extrinsic hallucinations from its parametric memory as it does not have access to an external knowledge base. Finally, the interactive feature of ChatGPT enables human collaboration with the underlying LLM to improve its performance, i.e, 8% ROUGE-1 on summarization and 2% ChrF++ on machine translation, in a multi-turn "prompt engineering" fashion. We also release codebase for evaluation set extraction.

Key facts. Cited by 894 papers in the map (data through 13 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 80 of its 151 references that are in the map. First posted on arXiv 8 Feb 2023.

Builds on

80 of 151 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Emergent Analogical Reasoning in Large Language Models19 Dec

Other references · 79

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  3. 2022Training language models to follow instructions with human feedbackcited by 15,856
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,657
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,480
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,880
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  12. 2017Deep reinforcement learning from human preferencescited by 3,871
38 more
  1. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,716
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  5. 2022Emergent Abilities of Large Language Modelscited by 2,499
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,985
  7. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,805
  8. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,793
  9. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,302
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,219
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,201
  12. 2015Teaching Machines to Read and Comprehendcited by 1,045
  13. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 810
  14. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 792
  15. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  16. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 533
  17. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 478
  18. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 333
  19. 2022News Summarization and Evaluation in the Era of GPT-3cited by 326
  20. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 310
  21. 2019Abductive Commonsense Reasoningcited by 300
  22. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  23. 2020MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselinescited by 203
  24. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  25. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  26. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 158
  27. 2021Explaining Answers with Entailment Treescited by 148
  28. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  29. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  30. 2020UBAR: Towards Fully End-to-End Task-Oriented Dialog Systems with GPT-2cited by 111
  31. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 108
  32. 2020Knowledge-Grounded Dialogue Generation with Pre-trained Language Modelscited by 107
  33. 2022ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reportscited by 101
  34. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  35. 2022ChatGPT: The End of Online Exam Integrity?cited by 88
  36. 2023ChatGPT is not all you need. A State of the Art Review of large Generative AI modelscited by 86
  37. 2021SpartQA: : A Textual Question Answering Benchmark for Spatial Reasoningcited by 75
  38. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 380; 2024: 341; 2025: 118; 2026: 55.

894 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,559
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,196
  3. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,709
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,594
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,433
  6. 2023A Survey on Evaluation of Large Language Modelscited by 1,236
  7. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,001
  8. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 983
  9. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 697
  10. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 666
  11. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 616
  12. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 485
38 more
  1. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 442
  2. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 421
  3. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 400
  4. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  5. 2023An Embodied Generalist Agent in 3D Worldcited by 357
  6. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 342
  7. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 336
  8. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 334
  9. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 331
  10. 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 326
  11. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  12. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 310
  13. 2023Low-Resource Languages Jailbreak GPT-4cited by 306
  14. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  15. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 280
  16. 2023How is ChatGPT's behavior changing over time?cited by 268
  17. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 264
  18. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
  19. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 253
  20. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 249
  21. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  22. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 241
  23. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 236
  24. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 235
  25. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  26. 2023Query Rewriting for Retrieval-Augmented Large Language Modelscited by 212
  27. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 211
  28. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 205
  29. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  30. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 204
  31. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  32. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 201
  33. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 200
  34. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 198
  35. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 192
  36. 2024Corrective Retrieval Augmented Generationcited by 192
  37. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 190
  38. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189

50 most cited of 894 All 894 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Pushing the Limits of ChatGPT on NLP Tasks
  2. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate
  3. 2024Testing and Evaluation of Large Language Models: Correctness, Non-Toxicity, and Fairness
  4. 2023Consistency Analysis of ChatGPT
  5. 2023Beyond Text: Unveiling Multimodal Proficiency of Large Language Models with MultiAPI Benchmark
  6. 2023Is ChatGPT a Good Causal Reasoner? A Comprehensive Evaluation
  7. 2024WILT: A Multi-Turn, Memorization-Robust Inductive Logic Benchmark for LLMs
  8. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluators