mlsanityData to 7 Oct 2026

A Survey on the Real Power of ChatGPT

arXiv 2405.00704 ↗First posted 22 Apr 2024cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

ChatGPT has changed the AI community and an active research line is the performance evaluation of ChatGPT. A key challenge for the evaluation is that ChatGPT is still closed-source and traditional benchmark datasets may have been used by ChatGPT as the training data. In this paper, (i) we survey recent studies which uncover the real performance levels of ChatGPT in seven categories of NLP tasks, (ii) review the social implications and safety issues of ChatGPT, and (iii) emphasize key challenges and opportunities for its evaluation. We hope our survey can shed some light on its blackbox manner, so that researchers are not misleaded by its surface generation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 50 of its 65 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

50 of 65 references.

References · 50

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  5. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  6. 2023Large Language Modelscited by 774
  7. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  8. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  9. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  10. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  11. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  12. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
38 more
  1. 2023How is ChatGPT's behavior changing over time?cited by 265
  2. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  3. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  4. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  5. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  6. 2023Query2doc: Query Expansion with Large Language Modelscited by 146
  7. 2023Don't Trust ChatGPT when Your Question is not in English: A Study of Multilingual Abilities and Types of LLMscited by 103
  8. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 98
  9. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 92
  10. 2023DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Modelscited by 75
  11. 2023Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Searchcited by 74
  12. 2023Learning to Tokenize for Generative Retrievalcited by 66
  13. 2023Empirical Study of Zero-Shot NER with ChatGPTcited by 60
  14. 2023Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Taskscited by 59
  15. 2023ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfercited by 49
  16. 2023ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustnesscited by 41
  17. 2023ChatGPT for Zero-shot Dialogue State Tracking: A Solution or an Opportunity?cited by 38
  18. 2023Consistency Analysis of ChatGPTcited by 37
  19. 2023GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLPcited by 37
  20. 2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
  21. 2023Privacy Implications of Retrieval-Based Language Modelscited by 31
  22. 2023CodeFusion: A Pre-trained Diffusion Model for Code Generationcited by 31
  23. 2023Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answeringcited by 28
  24. 2023Primacy Effect of ChatGPTcited by 27
  25. 2023Towards LLM-driven Dialogue State Trackingcited by 25
  26. 2023Putting ChatGPT's Medical Advice to the (Turing) Testcited by 23
  27. 2023ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Modelscited by 23
  28. 2023Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformerscited by 22
  29. 2023Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPTcited by 16
  30. 2023Is ChatGPT a Good Multi-Party Conversation Solver?cited by 13
  31. 2023Can ChatGPT Perform Reasoning Using the IRAC Method in Analyzing Legal Scenarios Like a Lawyer?cited by 12
  32. 2023CP-BCS: Binary Code Summarization Guided by Control Flow Graph and Pseudo Codecited by 11
  33. 2023ChatGPT for Suicide Risk Assessment on Social Media: Quantitative Evaluation of Model Performance, Potentials and Limitationscited by 9
  34. 2023Multilingual Coarse Political Stance Classification of Media. The Editorial Line of a ChatGPT and Bard Newspapercited by 9
  35. 2023UZH_CLyp at SemEval-2023 Task 9: Head-First Fine-Tuning and ChatGPT Data Generation for Cross-Lingual Learning in Tweet Intimacy Predictioncited by 8
  36. 2023Battle of the Large Language Models: Dolly vs LLaMA vs Vicuna vs Guanaco vs Bard vs ChatGPT -- A Text-to-SQL Parsing Comparisoncited by 6
  37. 2022What to Read in a Contract? Party-Specific Summarization of Legal Obligations, Entitlements, and Prohibitionscited by 5
  38. 2023Breeding Machine Translations: Evolutionary approach to survive and thrive in the world of automated evaluationcited by 5

Built on by

Not cited yet.