mlsanityData to 14 Oct 2026

KptLLM++: Towards Generic Keypoint Comprehension with Large Language Model

arXiv 2507.11102 ↗First posted 15 Jul 2025cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The emergence of Multimodal Large Language Models (MLLMs) has revolutionized image understanding by bridging textual and visual modalities. However, these models often struggle with capturing fine-grained semantic information, such as the precise identification and analysis of object keypoints. Keypoints, as structure-aware, pixel-level, and compact representations of objects, particularly articulated ones, play a crucial role in applications such as fine-grained image analysis, object retrieval, and behavior recognition. In this paper, we propose KptLLM++, a novel multimodal large language model that specifically designed for generic keypoint comprehension through the integration of diverse input modalities guided by user-defined instructions. By unifying keypoint detection across varied contexts, KptLLM++ establishes itself as an advanced interface, fostering more effective human-AI collaboration. The model is built upon a novel identify-then-detect paradigm, which first interprets keypoint semantics and subsequently localizes their precise positions through a structured chain-of-thought reasoning mechanism. To push the boundaries of performance, we have scaled up the training dataset to over 500K samples, encompassing diverse objects, keypoint categories, image styles, and scenarios with complex occlusions. This extensive scaling enables KptLLM++ to unlock its potential, achieving remarkable accuracy and generalization. Comprehensive experiments on multiple keypoint detection benchmarks demonstrate its state-of-the-art performance, underscoring its potential as a unified solution for fine-grained image understanding and its transformative implications for human-AI interaction.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 85 of its 99 references that are in the map. First posted on arXiv 15 Jul 2025.

Builds on

85 of 99 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension4 Nov
  2. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks12 Jun
  3. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  4. 2023VILA: On Pre-training for Visual Language Models12 Dec
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 78

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Segment Anythingcited by 8,234
  9. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
38 more
  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2022A ConvNet for the 2020scited by 3,579
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  5. 2023Mistral 7Bcited by 3,164
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2019Deep High-Resolution Representation Learning for Human Pose Estimationcited by 1,339
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  12. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  13. 2016Stacked Hourglass Networks for Human Pose Estimationcited by 915
  14. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  15. 2023Instruction Tuning with GPT-4cited by 686
  16. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  17. 2018Simple Baselines for Human Pose Estimation and Trackingcited by 502
  18. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  19. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 456
  20. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  21. 2017Cascaded Pyramid Network for Multi-Person Pose Estimationcited by 341
  22. 2024What matters when building vision-language models?cited by 326
  23. 2016Convolutional Pose Machinescited by 325
  24. 2013DeepPose: Human Pose Estimation via Deep Neural Networkscited by 305
  25. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  26. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  27. 2019HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimationcited by 247
  28. 2020InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Imagecited by 245
  29. 2018CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmarkcited by 214
  30. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 206
  31. 2020Whole-Body Human Pose Estimation in the Wildcited by 178
  32. 2021HRFormer: High-Resolution Transformer for Dense Predictioncited by 149
  33. 2018Look at Boundary: A Boundary-Aware Face Alignment Algorithmcited by 138
  34. 2019DeepFashion2: A Versatile Benchmark for Detection, Pose Estimation, Segmentation and Re-Identification of Clothing Imagescited by 134
  35. 2021Human Pose Regression with Residual Log-likelihood Estimationcited by 131
  36. 2017Compositional Human Pose Regressioncited by 124
  37. 2021AP-10K: A Benchmark for Animal Pose Estimation in the Wildcited by 123
  38. 2023DetGPT: Detect What You Need via Reasoningcited by 112

50 most cited of 78 All 78 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  2. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  3. 2024ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models
  4. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  5. 2024NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Vision
  6. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge
  7. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Exploration
  8. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation