mlsanityData to 14 Oct 2026

FathomGPT: A Natural Language Interface for Interactively Exploring Ocean Science Data

arXiv 2412.02784 ↗First posted 3 Dec 2024cs.HC cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

We introduce FathomGPT, an open source system for the interactive investigation of ocean science data via a natural language interface. FathomGPT was developed in close collaboration with marine scientists to enable researchers to explore and analyze the FathomNet image database. FathomGPT provides a custom information retrieval pipeline that leverages OpenAI's large language models to enable: the creation of complex queries to retrieve images, taxonomic information, and scientific measurements; mapping common names and morphological features to scientific names; generating interactive charts on demand; and searching by image or specified patterns within an image. In designing FathomGPT, particular emphasis was placed on enhancing the user's experience by facilitating free-form exploration and optimizing response times. We present an architectural overview and implementation details of FathomGPT, along with a series of ablation studies that demonstrate the effectiveness of our approach to name resolution, fine tuning, and prompt modification. We also present usage scenarios of interactive data exploration sessions and document feedback from ocean scientists and machine learning experts.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 33 of its 53 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

33 of 53 references.

References · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  9. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  10. 2023Segment Anythingcited by 8,234
  11. 2016Densely Connected Convolutional Networkscited by 6,720
  12. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
21 more
  1. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  2. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  3. 2019Language Models as Knowledge Bases?cited by 1,936
  4. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  5. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  6. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  7. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  8. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  9. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 477
  10. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 385
  11. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 187
  12. 2015Large-scale Simple Question Answering with Memory Networkscited by 127
  13. 2019A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualizationcited by 88
  14. 2023ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Languagecited by 73
  15. 2023Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Modelscited by 67
  16. 2023Graphologue: Exploring Large Language Model Responses with Interactive Diagramscited by 66
  17. 2021FathomNet: A global image database for enabling artificial intelligence in the oceancited by 25
  18. 2020Knowledge Graph-based Question Answering with Electronic Health Recordscited by 19
  19. 2023GenAssist: Making Image Generation Accessiblecited by 18
  20. 2021The Animal ID Problem: Continual Curationcited by 5
  21. 2023Designing Ocean Vision AI: An Investigation of Community Needs for Imaging-based Ocean Conservationcited by 3

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025HealthGenie: Empowering Users with Healthy Dietary Guidance through Knowledge Graph and Large Language Modelscited by 4
  2. 2025OKG-LLM: Aligning Ocean Knowledge Graph with Observation Data via LLMs for Global Sea Surface Temperature Predictioncited by 4
  3. 2025LitLinker: Supporting the Ideation of Interdisciplinary Contexts with Large Language Models for Teaching Literature in Elementary Schoolscited by 0