# Benchmarking Large Language Models for Image Classification of Marine Mammals

arXiv: https://arxiv.org/abs/2410.19848 · HTML: https://mlsanity.com/paper/2410.19848/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 40 references that are in the map.
- First posted on arXiv 22 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks](https://mlsanity.com/paper/2403.04783/) (2024)
- [Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation](https://mlsanity.com/paper/2310.01320/) (2023)
- [Embodied LLM Agents Learn to Cooperate in Organized Teams](https://mlsanity.com/paper/2403.12482/) (2024)
- [LLMGeo: Benchmarking Large Language Models on Image Geolocation In-the-wild](https://mlsanity.com/paper/2405.20363/) (2024)
- [Enhanced Detection Classification via Clustering SVM for Various Robot Collaboration Task](https://mlsanity.com/paper/2405.03026/) (2024)
- [Evaluating Zero-Shot GPT-4V Performance on 3D Visual Question Answering Benchmarks](https://mlsanity.com/paper/2405.18831/) (2024)
- [Enhancing Text Authenticity: A Novel Hybrid Approach for AI-Generated Text Detection](https://mlsanity.com/paper/2406.06558/) (2024)

## Built on by (most cited first, 4 of 4)

- [CrowdVLM-R1: Expanding R1 Ability to Vision Language Model for Crowd Counting using Fuzzy Group Relative Policy Reward](https://mlsanity.com/paper/2504.03724/) (2025)
- [Coverage-Aware Web Crawling for Domain-Specific Supplier Discovery via a Web--Knowledge--Web Pipeline](https://mlsanity.com/paper/2602.24262/) (2026)
- [When Corrective Hints Hurt: Prompt Design in Reasoner-Guided Repair of LLM Overcaution on Entailed Negations under OWL~2~DL](https://mlsanity.com/paper/2604.23398/) (2026)
- [When Large Language Models Do Not Work: Online Incivility Prediction through Graph Neural Networks](https://mlsanity.com/paper/2512.07684/) (2025)

## Abstract (arXiv)

As Artificial Intelligence (AI) has developed rapidly over the past few decades, the new generation of AI, Large Language Models (LLMs) trained on massive datasets, has achieved ground-breaking performance in many applications. Further progress has been made in multimodal LLMs, with many datasets created to evaluate LLMs with vision abilities. However, none of those datasets focuses solely on marine mammals, which are indispensable for ecological equilibrium. In this work, we build a benchmark dataset with 1,423 images of 65 kinds of marine mammals, where each animal is uniquely classified into different levels of class, ranging from species-level to medium-level to group-level. Moreover, we evaluate several approaches for classifying these marine mammals: (1) machine learning (ML) algorithms using embeddings provided by neural networks, (2) influential pre-trained neural networks, (3) zero-shot models: CLIP and LLMs, and (4) a novel LLM-based multi-agent system (MAS). The results demonstrate the strengths of traditional models and LLMs in different aspects, and the MAS can further improve the classification performance. The dataset is available on GitHub: https://github.com/yeyimilk/LLM-Vision-Marine-Animals.git.