# Learning Class Difficulty via Dynamic Focal Attention for Histopathology Segmentation

arXiv: https://arxiv.org/abs/2604.13479 · HTML: https://mlsanity.com/paper/2604.13479/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 15 Apr 2026.

## Builds on (most cited first, 17 of 17)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Unified Focal loss: Generalising Dice and cross entropy-based losses to handle class imbalanced medical image segmentation](https://mlsanity.com/paper/2102.04525/) (2021)
- [MILD-Net: Minimal Information Loss Dilated Network for Gland Instance Segmentation in Colon Histology Images](https://mlsanity.com/paper/1806.01963/) (2018)
- [CoNIC Challenge: Pushing the Frontiers of Nuclear Detection, Segmentation, Classification and Counting](https://mlsanity.com/paper/2303.06274/) (2023)
- [SAM-Path: A Segment Anything Model for Semantic Segmentation in Digital Pathology](https://mlsanity.com/paper/2307.09570/) (2023)

## Abstract (arXiv)

Frequency-based loss reweighting, the standard remedy for imbalanced histopathology segmentation, implicitly assumes that rare classes are difficult. Yet difficulty also arises from morphological variability, boundary ambiguity, and contextual similarity, all largely orthogonal to class frequency. We propose Dynamic Focal Attention (DFA), a simple, efficient mechanism that learns class-specific difficulty directly within the cross-attention of query-based mask decoders. DFA adds a learnable per-class bias to the attention logits, reweighting representations before prediction rather than gradients after it. Initialised from a centred log-frequency prior to prevent gradient starvation and then optimised end-to-end, the bias adapts to difficulty signals as they emerge during training, unifying frequency- and difficulty-aware reweighting in a single attention-bias framework. On three benchmarks (BCSS, BDSA, CRAG), DFA consistently improves Dice and IoU, matching or exceeding a two-stage difficulty-aware baseline without a separate estimator or extra training stage. This shows that encoding difficulty at the representation level is a principled alternative to conventional loss reweighting.