# Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency

arXiv: https://arxiv.org/abs/2407.09519 · HTML: https://mlsanity.com/paper/2407.09519/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 34 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Builds on (most cited first, 8 of 8)

- [Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection](https://mlsanity.com/paper/2309.12247/) (2023)
- [ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishing](https://mlsanity.com/paper/2303.13367/) (2023)
- [Unveiling the Safety of GPT-4o: An Empirical Study using Jailbreak Attacks](https://mlsanity.com/paper/2406.06302/) (2024)
- [GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot Setting and Performance Boosting Through Prompts](https://mlsanity.com/paper/2305.12477/) (2023)
- [GAN Computers Generate Arts? A Survey on Visual Arts, Music, and Literary Text Generation using Generative Adversarial Network](https://mlsanity.com/paper/2108.03857/) (2021)
- [M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Models](https://mlsanity.com/paper/2405.15638/) (2024)
- [CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations](https://mlsanity.com/paper/2109.00181/) (2021)
- [The Imitation Game: Detecting Human and AI-Generated Texts in the Era of ChatGPT and BARD](https://mlsanity.com/paper/2307.12166/) (2023)

## Built on by (most cited first, 30 of 31)

- [A Comprehensive Evaluation of Cognitive Biases in LLMs](https://mlsanity.com/paper/2410.15413/) (2024)
- [SafeEmbodAI: a Safety Framework for Mobile Robots in Embodied AI Systems](https://mlsanity.com/paper/2409.01630/) (2024)
- [Addressing Bias in Generative AI: Challenges and Research Opportunities in Information Management](https://mlsanity.com/paper/2502.10407/) (2025)
- [3DAxisPrompt: Promoting the 3D Grounding and Reasoning in GPT-4o](https://mlsanity.com/paper/2503.13185/) (2025)
- [Labeling Free-text Data using Language Model Ensembles](https://mlsanity.com/paper/2501.08413/) (2025)
- [CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methods](https://mlsanity.com/paper/2508.18753/) (2025)
- [Learning to Plan Before Answering: Self-Teaching LLMs to Learn Abstract Plans for Problem Solving](https://mlsanity.com/paper/2505.00031/) (2025)
- [LLM Online Spatial-temporal Signal Reconstruction Under Noise](https://mlsanity.com/paper/2411.15764/) (2024)
- [Exploring Multimodal Perception in Large Language Models Through Perceptual Strength Ratings](https://mlsanity.com/paper/2503.06980/) (2025)
- [From GPT-3 to GPT-5: Mapping their capabilities, scope, limitations, and consequences](https://mlsanity.com/paper/2604.10332/) (2026)
- [Test-Time Alignment for Large Language Models via Textual Model Predictive Control](https://mlsanity.com/paper/2502.20795/) (2025)
- [Recursive Offloading for LLM Serving in Multi-tier Networks](https://mlsanity.com/paper/2505.16502/) (2025)
- [Scatter-Based Innovation Propagation in Large Language Models for Multi-Stage Process Adaptation](https://mlsanity.com/paper/2506.17949/) (2025)
- [The Bots of Persuasion: Examining How Conversational Agents' Linguistic Expressions of Personality Affect User Perceptions and Decisions](https://mlsanity.com/paper/2602.17185/) (2026)
- [GPT Sonograpy: Hand Gesture Decoding from Forearm Ultrasound Images via VLM](https://mlsanity.com/paper/2407.10870/) (2024)
- [From Predictive Importance to Causality: Which Machine Learning Model Reflects Reality?](https://mlsanity.com/paper/2409.02130/) (2024)
- [Predicting User Stances from Target-Agnostic Information using Large Language Models](https://mlsanity.com/paper/2409.14395/) (2024)
- [DeepSeek on a Trip: Inducing Targeted Visual Hallucinations via Representation Vulnerabilities](https://mlsanity.com/paper/2502.07905/) (2025)
- [AI University: An LLM-Powered Learning Assistant for Engineering---A Finite Element Method Case Study](https://mlsanity.com/paper/2504.08846/) (2025)
- [Ratas framework: A comprehensive genai-based approach to rubric-based marking of real-world textual exams](https://mlsanity.com/paper/2505.23818/) (2025)
- [An Empirical Study of Bugs in Data Visualization Libraries](https://mlsanity.com/paper/2506.15084/) (2025)
- [MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning](https://mlsanity.com/paper/2507.21924/) (2025)
- [Enhancing Reliability in LLM-Integrated Robotic Systems: A Unified Approach to Security and Safety](https://mlsanity.com/paper/2509.02163/) (2025)
- [From "Help" to Helpful: A Hierarchical Assessment of LLMs in Mental e-Health Applications](https://mlsanity.com/paper/2602.18443/) (2026)
- [Parse, Search, and Confirmation: Training-Free Aerial Vision-and-Dialog Navigation with Chain-of-Thought Reasoning and Structured Spatial Memory](https://mlsanity.com/paper/2607.11529/) (2026)
- [A Dynamic and High-Precision Method for Scenario-Based HRA Synthetic Data Collection in Multi-Agent Collaborative Environments Driven by LLMs](https://mlsanity.com/paper/2502.00022/) (2025)
- [Context-Aware Scientific Knowledge Extraction on Linked Open Data using Large Language Models](https://mlsanity.com/paper/2506.17580/) (2025)
- [Exploring Decision-Making Capabilities of LLM Agents: An Experimental Study on Jump-Jump Game](https://mlsanity.com/paper/2509.00483/) (2025)
- [To See or To Read: User Behavior Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2511.03845/) (2025)
- [Instruction Tuning of Large Language Models for Tabular Data Generation-in One Day](https://mlsanity.com/paper/2511.23220/) (2025)

## Abstract (arXiv)

As large language models (LLMs) continue to advance, evaluating their comprehensive capabilities becomes significant for their application in various fields. This research study comprehensively evaluates the language, vision, speech, and multimodal capabilities of GPT-4o. The study employs standardized exam questions, reasoning tasks, and translation assessments to assess the model's language capability. Additionally, GPT-4o's vision and speech capabilities are tested through image classification and object recognition tasks, as well as accent classification. The multimodal evaluation assesses the model's performance in integrating visual and linguistic data. Our findings reveal that GPT-4o demonstrates high accuracy and efficiency across multiple domains in language and reasoning capabilities, excelling in tasks that require few-shot learning. GPT-4o also provides notable improvements in multimodal tasks compared to its predecessors. However, the model shows variability and faces limitations in handling complex and ambiguous inputs, particularly in audio and vision capabilities. This paper highlights the need for more comprehensive benchmarks and robust evaluation frameworks, encompassing qualitative assessments involving human judgment as well as error analysis. Future work should focus on expanding datasets, investigating prompt-based assessment, and enhancing few-shot learning techniques to test the model's practical applicability and performance in real-world scenarios.