mlsanityData to 7 Oct 2026

GPT Sonograpy: Hand Gesture Decoding from Forearm Ultrasound Images via VLM

arXiv 2407.10870 ↗First posted 15 Jul 2024cs.CV cs.AI cs.HCcited by 1

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Large vision-language models (LVLMs), such as the Generative Pre-trained Transformer 4-omni (GPT-4o), are emerging multi-modal foundation models which have great potential as powerful artificial-intelligence (AI) assistance tools for a myriad of applications, including healthcare, industrial, and academic sectors. Although such foundation models perform well in a wide range of general tasks, their capability without fine-tuning is often limited in specialized tasks. However, full fine-tuning of large foundation models is challenging due to enormous computation/memory/dataset requirements. We show that GPT-4o can decode hand gestures from forearm ultrasound data even with no fine-tuning, and improves with few-shot, in-context learning.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 38 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

20 of 38 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models6 Jul

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  9. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  10. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  11. 2022Images Speak in Images: A Generalist Painter for In-Context Visual Learningcited by 317
  12. 2022Visual Prompting via Image Inpaintingcited by 248
7 more
  1. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  2. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  3. 2024Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiencycited by 31
  4. 2023Instruct Me More! Random Prompting for Visual In-Context Learningcited by 30
  5. 2023Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoningcited by 22
  6. 2022Simultaneous Estimation of Hand Configurations and Finger Joint Angles using Forearm Ultrasoundcited by 7
  7. 2022FashionVQA: A Domain-Specific Visual Question Answering Systemcited by 6

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Human Re-ID Meets LVLMs: What can we expect?cited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Enhancing Medical Task Performance in GPT-4V: A Comprehensive Study on Prompt Engineering Strategies
  2. 2025Robust Photo-Realistic Hand Gesture Generation: from Single View to Multiple View
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  4. 2024An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging
  5. 2023Holistic Evaluation of GPT-4V for Biomedical Imaging
  6. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction
  7. 2025FETAL-GAUGE: A Benchmark for Assessing Vision-Language Models in Fetal Ultrasound
  8. 2025U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding