mlsanityData to 14 Oct 2026

MouseGPT: A Large-scale Vision-Language Model for Mouse Behavior Analysis

arXiv 2503.10212 ↗First posted 13 Mar 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Analyzing animal behavior is crucial in advancing neuroscience, yet quantifying and deciphering its intricate dynamics remains a significant challenge. Traditional machine vision approaches, despite their ability to detect spontaneous behaviors, fall short due to limited interpretability and reliance on manual labeling, which restricts the exploration of the full behavioral spectrum. Here, we introduce MouseGPT, a Vision-Language Model (VLM) that integrates visual cues with natural language to revolutionize mouse behavior analysis. Built upon our first-of-its-kind dataset - incorporating pose dynamics and open-vocabulary behavioral annotations across over 42 million frames of diverse psychiatric conditions - MouseGPT provides a novel, context-rich method for comprehensive behavior interpretation. Our holistic analysis framework enables detailed behavior profiling, clustering, and novel behavior discovery, offering deep insights without the need for labor - intensive manual annotation. Evaluations reveal that MouseGPT surpasses existing models in precision, adaptability, and descriptive richness, positioning it as a transformative tool for ethology and for unraveling complex behavioral dynamics in animal models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 8 of its 61 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

8 of 61 references.

References · 8

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2024Qwen2 Technical Reportcited by 2,120
  5. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,420
  6. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  7. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  8. 2019Gradio: Hassle-Free Sharing and Testing of ML Models in the Wildcited by 178

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026BehaviorVLM: Unified Finetuning-Free Behavioral Understanding with Vision-Language Reasoningcited by 0
  2. 2026MTT-Bench: Predicting Social Dominance in Mice via Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Preliminary Use of Vision Language Model Driven Extraction of Mouse Behavior Towards Understanding Fear Expression
  2. 2023AmadeusGPT: a natural language interface for interactive animal behavioral analysis
  3. 2026Rodent-Bench
  4. 2024GPT-4o: Visual perception performance of multimodal large language models in piglet activity understanding
  5. 2026CheeseBench: Evaluating Large Language Models on Rodent Behavioral Neuroscience Paradigms
  6. 2025AnimalMotionCLIP: Embedding motion in CLIP for Animal Behavior Analysis
  7. 2025NLP4Neuro: Sequence-to-sequence learning for neural population decoding
  8. 2026Integrative neurocybernetic modeling in the era of large-scale neuroscience