# Learn Fine-grained Adaptive Loss for Multiple Anatomical Landmark Detection in Medical Images

arXiv: https://arxiv.org/abs/2105.09124 · HTML: https://mlsanity.com/paper/2105.09124/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 19 May 2021.

## Builds on (most cited first, 26 of 26)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Simple Baselines for Human Pose Estimation and Tracking](https://mlsanity.com/paper/1804.06208/) (2018)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [Cascaded Pyramid Network for Multi-Person Pose Estimation](https://mlsanity.com/paper/1711.07319/) (2017)
- [Designing Neural Network Architectures using Reinforcement Learning](https://mlsanity.com/paper/1611.02167/) (2016)
- [A Survey of the Recent Architectures of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1901.06032/) (2019)
- [Unsupervised Learning of Object Landmarks through Conditional Image Generation](https://mlsanity.com/paper/1806.07823/) (2018)
- [Integrating Spatial Configuration into Heatmap Regression Based CNNs for Landmark Localization](https://mlsanity.com/paper/1908.00748/) (2019)
- [Flowing ConvNets for Human Pose Estimation in Videos](https://mlsanity.com/paper/1506.02897/) (2015)
- [Adversarial PoseNet: A Structure-aware Convolutional Network for Human Pose Estimation](https://mlsanity.com/paper/1705.00389/) (2017)
- [Cephalometric Landmark Detection by AttentiveFeature Pyramid Fusion and Regression-Voting](https://mlsanity.com/paper/1908.08841/) (2019)
- [AM-LFS: AutoML for Loss Function Search](https://mlsanity.com/paper/1905.07375/) (2019)
- [Feature Boosting Network For 3D Pose Estimation](https://mlsanity.com/paper/1901.04877/) (2019)
- [Deep Learning-Based Regression and Classification for Automatic Landmark Localization in Medical Images](https://mlsanity.com/paper/2007.05295/) (2020)
- [An Empirical Study into Annotator Agreement, Ground Truth Estimation, and Algorithm Evaluation](https://mlsanity.com/paper/1307.0426/) (2013)
- [Facial Landmark Detection with Tweaked Convolutional Neural Networks](https://mlsanity.com/paper/1511.04031/) (2015)
- [Agent with Warm Start and Active Termination for Plane Localization in 3D Ultrasound](https://mlsanity.com/paper/1910.04331/) (2019)
- [Less is More: Simultaneous View Classification and Landmark Detection for Abdominal Ultrasound Images](https://mlsanity.com/paper/1805.10376/) (2018)
- [Enhanced detection of fetal pose in 3D MRI by Deep Reinforcement Learning with physical structure priors on anatomy](https://mlsanity.com/paper/2007.08146/) (2020)
- [Adaloss: Adaptive Loss Function for Landmark Localization](https://mlsanity.com/paper/1908.01070/) (2019)

## Built on by (most cited first, 6 of 6)

- [FM-OSD: Foundation Model-Enabled One-Shot Detection of Anatomical Landmarks](https://mlsanity.com/paper/2407.05412/) (2024)
- [CANet: Channel Extending and Axial Attention Catching Network for Multi-structure Kidney Segmentation](https://mlsanity.com/paper/2208.05241/) (2022)
- [Deep Learning Techniques for Automatic Lateral X-ray Cephalometric Landmark Detection: Is the Problem Solved?](https://mlsanity.com/paper/2409.15834/) (2024)
- [Aortic root landmark localization with optimal transport loss for heatmap regression](https://mlsanity.com/paper/2407.04921/) (2024)
- [Zero-shot System for Automatic Body Region Detection for Volumetric CT and MR Images](https://mlsanity.com/paper/2602.08717/) (2026)
- [Rethinking Medical Landmark Localization with Prototype Learning-based Progressive Offset Correction](https://mlsanity.com/paper/2608.09182/) (2026)

## Abstract (arXiv)

Automatic and accurate detection of anatomical landmarks is an essential operation in medical image analysis with a multitude of applications. Recent deep learning methods have improved results by directly encoding the appearance of the captured anatomy with the likelihood maps (i.e., heatmaps). However, most current solutions overlook another essence of heatmap regression, the objective metric for regressing target heatmaps and rely on hand-crafted heuristics to set the target precision, thus being usually cumbersome and task-specific. In this paper, we propose a novel learning-to-learn framework for landmark detection to optimize the neural network and the target precision simultaneously. The pivot of this work is to leverage the reinforcement learning (RL) framework to search objective metrics for regressing multiple heatmaps dynamically during the training process, thus avoiding setting problem-specific target precision. We also introduce an early-stop strategy for active termination of the RL agent's interaction that adapts the optimal precision for separate targets considering exploration-exploitation tradeoffs. This approach shows better stability in training and improved localization accuracy in inference. Extensive experimental results on two different applications of landmark localization: 1) our in-house prenatal ultrasound (US) dataset and 2) the publicly available dataset of cephalometric X-Ray landmark detection, demonstrate the effectiveness of our proposed method. Our proposed framework is general and shows the potential to improve the efficiency of anatomical landmark detection.