mlsanityData to 14 Oct 2026

KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension

arXiv 2411.01846 ↗First posted 4 Nov 2024cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent advancements in Multimodal Large Language Models (MLLMs) have greatly improved their abilities in image understanding. However, these models often struggle with grasping pixel-level semantic details, e.g., the keypoints of an object. To bridge this gap, we introduce the novel challenge of Semantic Keypoint Comprehension, which aims to comprehend keypoints across different task scenarios, including keypoint semantic understanding, visual prompt-based keypoint detection, and textual prompt-based keypoint detection. Moreover, we introduce KptLLM, a unified multimodal model that utilizes an identify-then-detect strategy to effectively address these challenges. KptLLM underscores the initial discernment of semantics in keypoints, followed by the precise determination of their positions through a chain-of-thought process. With several carefully designed modules, KptLLM adeptly handles various modality inputs, facilitating the interpretation of both semantic contents and keypoint locations. Our extensive experiments demonstrate KptLLM's superiority in various keypoint detection benchmarks and its unique semantic capabilities in interpreting keypoints.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 76 of its 83 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

76 of 83 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun
  2. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  3. 2023VILA: On Pre-training for Visual Language Models12 Dec
  4. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models13 Oct
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 69

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2023Segment Anythingcited by 8,234
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2023Mistral 7Bcited by 3,164
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  8. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  11. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  12. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 915
  13. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  14. 2023Instruction Tuning with GPT-4cited by 686
  15. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  16. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  17. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  18. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 456
  19. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  20. 2017Cascaded Pyramid Network for Multi-Person Pose Estimationcited by 341
  21. 2024What matters when building vision-language models?cited by 326
  22. 2016Convolutional Pose Machinescited by 325
  23. 2013DeepPose: Human Pose Estimation via Deep Neural Networkscited by 305
  24. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  25. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  26. 2019HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimationcited by 247
  27. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
  28. 2018CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmarkcited by 214
  29. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 206
  30. 2020Whole-Body Human Pose Estimation in the Wildcited by 178
  31. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  32. 2018Look at Boundary: A Boundary-Aware Face Alignment Algorithmcited by 138
  33. 2019DeepFashion2: A Versatile Benchmark for Detection, Pose Estimation, Segmentation and Re-Identification of Clothing Imagescited by 134
  34. 2021Human Pose Regression with Residual Log-likelihood Estimationcited by 131
  35. 2017Compositional Human Pose Regressioncited by 124
  36. 2021AP-10K: A Benchmark for Animal Pose Estimation in the Wildcited by 123
  37. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  38. 2019Cross-Domain Adaptation for Animal Pose Estimationcited by 108

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 4.

5 papers.

  1. 2024GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Groundingcited by 28
  2. 2025Recurrent Feature Mining and Keypoint Mixup Padding for Category-Agnostic Pose Estimationcited by 2
  3. 2025Estimating 2D Keypoints of Surgical Tools Using Vision-Language Models with Low-Rank Adaptationcited by 1
  4. 2025KptLLM++: Towards Generic Keypoint Comprehension with Large Language Modelcited by 0
  5. 2025Talking Points: Describing and Localizing Pixelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models
  2. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts
  3. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs
  4. 2024OpenKD: Opening Prompt Diversity for Zero- and Few-shot Keypoint Detection
  5. 2026Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding
  6. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  7. 2026When Prompts Become Pixels: Prompt-Region Grounding for Multimodal Reasoning
  8. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge