# J-EDI QA: Benchmark for deep-sea organism-specific multimodal LLM

arXiv: https://arxiv.org/abs/2412.15574 · HTML: https://mlsanity.com/paper/2412.15574/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 6 references that are in the map.
- First posted on arXiv 20 Dec 2024.

## Builds on (most cited first, 6 of 6)

- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [Perception Test: A Diagnostic Benchmark for Multimodal Video Models](https://mlsanity.com/paper/2305.13786/) (2023)

## Built on by (most cited first, 1 of 1)

- [Earth Science Foundation Models: From Perception to Reasoning and Discovery](https://mlsanity.com/paper/2605.12542/) (2026)

## Abstract (arXiv)

Japan Agency for Marine-Earth Science and Technology (JAMSTEC) has made available the JAMSTEC Earth Deep-sea Image (J-EDI), a deep-sea video and image archive (https://www.godac.jamstec.go.jp/jedi/e/index.html). This archive serves as a valuable resource for researchers and scholars interested in deep-sea imagery. The dataset comprises images and videos of deep-sea phenomena, predominantly of marine organisms, but also of the seafloor and physical processes. In this study, we propose J-EDI QA, a benchmark for understanding images of deep-sea organisms using a multimodal large language model (LLM). The benchmark is comprised of 100 images, accompanied by questions and answers with four options by JAMSTEC researchers for each image. The QA pairs are provided in Japanese, and the benchmark assesses the ability to understand deep-sea species in Japanese. In the evaluation presented in this paper, OpenAI o1 achieved a 50% correct response rate. This result indicates that even with the capabilities of state-of-the-art models as of December 2024, deep-sea species comprehension is not yet at an expert level. Further advances in deep-sea species-specific LLMs are therefore required.