mlsanityData to 14 Oct 2026

An In-depth Look at Gemini's Language Abilities

arXiv 2312.11444 ↗First posted 18 Dec 2023cs.CL cs.AIcited by 28

Not on the map. Why?

Abstract (arXiv)

The recently released Google Gemini class of models are the first to comprehensively report results that rival the OpenAI GPT series across a wide variety of tasks. In this paper, we do an in-depth exploration of Gemini's language abilities, making two contributions. First, we provide a third-party, objective comparison of the abilities of the OpenAI GPT and Google Gemini models with reproducible code and fully transparent results. Second, we take a closer look at the results, identifying areas where one of the two model classes excels. We perform this analysis over 10 datasets testing a variety of language abilities, including reasoning, answering knowledge-based questions, solving math problems, translating between languages, generating code, and acting as instruction-following agents. From this analysis, we find that Gemini Pro achieves accuracy that is close but slightly inferior to the corresponding GPT 3.5 Turbo on all tasks that we benchmarked. We further provide explanations for some of this under-performance, including failures in mathematical reasoning with many digits, sensitivity to multiple-choice answer ordering, aggressive content filtering, and others. We also identify areas where Gemini demonstrates comparably high performance, including generation into non-English languages, and handling longer and more complex reasoning chains. Code and data for reproduction can be found at https://github.com/neulab/gemini-benchmark

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 18 Dec 2023.

Builds on

25 of 34 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
13 more
  1. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  2. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  3. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  4. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  5. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 318
  6. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  7. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  8. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
  9. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  10. 2022Execution-Based Evaluation for Open-Domain Code Generationcited by 91
  11. 2023Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learningcited by 33
  12. 2023How to Design Translation Prompts for ChatGPT: An Empirical Studycited by 31
  13. 2023ChatGPT MT: Competitive for High- (but not Low-) Resource Languagescited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 18; 2025: 5; 2026: 2.

28 papers.

Sort:

By area:

  1. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  2. 2024Many-Shot In-Context Learningcited by 206
  3. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 174
  4. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  5. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 93
  6. 2025dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Modelcited by 47
  7. 2024Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMscited by 40
  8. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competitioncited by 18
  9. 2024Are Large Language Models More Empathetic than Humans?cited by 18
  10. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Modelscited by 16
  11. 2024Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settingscited by 14
  12. 2023Gemini Pro Defeated by GPT-4V: Evidence from Educationcited by 12
16 more
  1. 2024ChatNVD: Advancing Cybersecurity Vulnerability Assessment with Large Language Modelscited by 11
  2. 2025Deterministic or probabilistic? The psychology of LLMs as random number generatorscited by 7
  3. 2024Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settingscited by 6
  4. 2024MiTTenS: A Dataset for Evaluating Gender Mistranslationcited by 5
  5. 2024Self-Imagine: Effective Unimodal Reasoning with Multimodal Models using Self-Imaginationcited by 5
  6. 2024Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmerscited by 5
  7. 2024Language Modelling Approaches to Adaptive Machine Translationcited by 3
  8. 2024Evaluating the Efficacy of Prompt-Engineered Large Multimodal Models Versus Fine-Tuned Vision Transformers in Image-Based Security Applicationscited by 3
  9. 2025Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Drivingcited by 3
  10. 2024MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Modelscited by 2
  11. 2025An evaluation of LLMs and Google Translate for translation of selected Indian languages via sentiment and semantic analysescited by 2
  12. 2025Large Language Models Could Be Rote Learnerscited by 2
  13. 2024Leveraging LLMs for MT in Crisis Scenarios: a blueprint for low-resource languagescited by 1
  14. 2026Representational Similarity Alignment of Figurative Language in Humans and Large Language Modelscited by 1
  15. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Modelscited by 0
  16. 2026No Universal Courtesy: A Cross-Linguistic, Multi-Model Study of Politeness Effects on LLMs Using the PLUM Corpuscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases
  2. 2025Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospects
  3. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise
  4. 2025Evaluation of LLMs for mathematical problem solving
  5. 2025ChatGPT and Gemini participated in the Korean College Scholastic Ability Test -- Earth Science I
  6. 2024Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models
  7. 2026MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation
  8. 2025MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning