# LSP-DETR: Efficient and Scalable Nuclei Segmentation in Whole-Slide Images

arXiv: https://arxiv.org/abs/2601.03163 · HTML: https://mlsanity.com/paper/2601.03163/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 61 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [HoVer-Net: Simultaneous Segmentation and Classification of Nuclei in Multi-Tissue Histology Images](https://mlsanity.com/paper/1812.06499/) (2018)
- [Order Matters: Sequence to sequence for sets](https://mlsanity.com/paper/1511.06391/) (2015)
- [Cell Detection with Star-convex Polygons](https://mlsanity.com/paper/1806.03535/) (2018)
- [Fast Video Generation with Sliding Tile Attention](https://mlsanity.com/paper/2502.04507/) (2025)
- [PanNuke Dataset Extension, Insights and Baselines](https://mlsanity.com/paper/2003.10778/) (2020)
- [CellViT: Vision Transformers for Precise Cell Segmentation and Classification](https://mlsanity.com/paper/2306.15350/) (2023)
- [UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition](https://mlsanity.com/paper/2311.15599/) (2023)
- [Attention-Based Transformers for Instance Segmentation of Cells in Microstructures](https://mlsanity.com/paper/2011.09763/) (2020)
- [CPP-Net: Context-aware Polygon Proposal Network for Nucleus Segmentation](https://mlsanity.com/paper/2102.06867/) (2021)
- [Nuclei instance segmentation and classification in histopathology images with StarDist](https://mlsanity.com/paper/2203.02284/) (2022)
- [Mask-RCNN and U-net Ensembled for Nuclei Segmentation](https://mlsanity.com/paper/1901.10170/) (2019)
- [Learning the RoPEs: Better 2D and 3D Position Encodings with STRING](https://mlsanity.com/paper/2502.02562/) (2025)
- [Graph Neural Networks in Histopathology: Emerging Trends and Future Directions](https://mlsanity.com/paper/2406.12808/) (2024)
- [HoTPP Benchmark: Are We Good at the Long Horizon Events Forecasting?](https://mlsanity.com/paper/2406.14341/) (2024)
- [Affine-Consistent Transformer for Multi-Class Cell Nuclei Detection](https://mlsanity.com/paper/2310.14154/) (2023)
- [SAP-DETR: Bridging the Gap Between Salient Points and Queries-Based Transformer Detector for Fast Model Convergency](https://mlsanity.com/paper/2211.02006/) (2022)
- [LKCell: Efficient Cell Nuclei Instance Segmentation with Large Convolution Kernels](https://mlsanity.com/paper/2407.18054/) (2024)
- [A Comprehensive Overview of Computational Nuclei Segmentation Methods in Digital Pathology](https://mlsanity.com/paper/2308.08112/) (2023)

## Built on by (most cited first, 1 of 1)

- [Comprehensive Evaluation and Fine-Tuning of Foundational Cell Nuclei Segmentation Models in Renal Pathology](https://mlsanity.com/paper/2610.07711/) (2026)

## Abstract (arXiv)

Background and Objective: Precise and scalable instance segmentation of cell nuclei is a fundamental prerequisite for computational pathology, yet gigapixel whole-slide images (WSIs) pose significant computational challenges. While patch-based processing is standard during training, existing methods are often limited to small tile sizes during inference due to architectural bottlenecks or reliance on computationally expensive post-processing for instance separation. We introduce a faster, scalable, and end-to-end framework capable of processing large-scale image tiles while accurately modeling biologically realistic overlapping nuclei. Methods: We propose LSP-DETR (Local Star Polygon DEtection TRansformer). The model represents nuclei as star-convex polygons and employs a lightweight transformer with linear complexity, enabling the processing of high-resolution images in a single forward pass. A novel radial distance loss accommodates annotation uncertainty, allowing the segmentation of overlapping nuclei to emerge naturally without explicit overlap labels. Results: LSP-DETR achieves state-of-the-art efficiency, with an inference time of 0.45 s/mm2, a 3.2x speedup over StarDist, the next-fastest method. On PanNuke, the model achieves competitive accuracy (67.5 bPQ), while yielding an F1-score of 0.964 in polygon overlap when evaluated against consensus annotations from two expert pathologists. Furthermore, it outperforms larger models such as LKCell in generalization robustness, reaching an F1-score of 85.0 on MoNuSeg. Conclusions: LSP-DETR bridges the gap between high-fidelity segmentation and practical clinical requirements by eliminating heuristic post-processing. By providing a scalable, linear-complexity solution that naturally handles overlaps between nuclei, this framework sets a new direction for efficient high-throughput WSI analysis in digital pathology.