# ChatGPT vs Gemini vs LLaMA on Multilingual Sentiment Analysis

arXiv: https://arxiv.org/abs/2402.01715 · HTML: https://mlsanity.com/paper/2402.01715/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Jan 2024.

## Built on by (most cited first, 13 of 13)

- [FAIRGAME: a Framework for AI Agents Bias Recognition using Game Theory](https://mlsanity.com/paper/2504.14325/) (2025)
- [Media and responsible AI governance: a game-theoretic and LLM analysis](https://mlsanity.com/paper/2503.09858/) (2025)
- [Controllable Text Generation in the Instruction-Tuning Era](https://mlsanity.com/paper/2405.01490/) (2024)
- [Silent Signals, Loud Impact: LLMs for Word-Sense Disambiguation of Coded Dog Whistles](https://mlsanity.com/paper/2406.06840/) (2024)
- [Evaluating Research Quality with Large Language Models: An Analysis of ChatGPT's Effectiveness with Different Settings and Inputs](https://mlsanity.com/paper/2408.06752/) (2024)
- [Zero-Shot Topic Classification of Column Headers: Leveraging LLMs for Metadata Enrichment](https://mlsanity.com/paper/2403.00884/) (2024)
- [Large Language Models' Detection of Political Orientation in Newspapers](https://mlsanity.com/paper/2406.00018/) (2024)
- [RogueGPT: dis-ethical tuning transforms ChatGPT4 into a Rogue AI in 158 Words](https://mlsanity.com/paper/2407.15009/) (2024)
- [Mind the Language Gap: Automated and Augmented Evaluation of Bias in LLMs for High- and Low-Resource Languages](https://mlsanity.com/paper/2504.18560/) (2025)
- [A Misleading Gallery of Fluid Motion by Generative Artificial Intelligence](https://mlsanity.com/paper/2405.15406/) (2024)
- [Trick or Neat: Adversarial Ambiguity and Language Model Evaluation](https://mlsanity.com/paper/2506.01205/) (2025)
- [Multimodal Generative AI with Autoregressive LLMs for Human Motion Understanding and Generation: A Way Forward](https://mlsanity.com/paper/2506.03191/) (2025)
- [Data-Efficient Adaptation and a Novel Evaluation Method for Aspect-based Sentiment Analysis](https://mlsanity.com/paper/2511.03034/) (2025)

## Abstract (arXiv)

Automated sentiment analysis using Large Language Model (LLM)-based models like ChatGPT, Gemini or LLaMA2 is becoming widespread, both in academic research and in industrial applications. However, assessment and validation of their performance in case of ambiguous or ironic text is still poor. In this study, we constructed nuanced and ambiguous scenarios, we translated them in 10 languages, and we predicted their associated sentiment using popular LLMs. The results are validated against post-hoc human responses. Ambiguous scenarios are often well-coped by ChatGPT and Gemini, but we recognise significant biases and inconsistent performance across models and evaluated human languages. This work provides a standardised methodology for automated sentiment analysis evaluation and makes a call for action to further improve the algorithms and their underlying data, to improve their performance, interpretability and applicability.