See Detail Say Clear: Towards Brain CT Report Generation via Pathological Clue-driven Representation Learning
Not on the map. Why?
Abstract (arXiv)
Brain CT report generation is significant to aid physicians in diagnosing cranial diseases. Recent studies concentrate on handling the consistency between visual and textual pathological features to improve the coherence of report. However, there exist some challenges: 1) Redundant visual representing: Massive irrelevant areas in 3D scans distract models from representing salient visual contexts. 2) Shifted semantic representing: Limited medical corpus causes difficulties for models to transfer the learned textual representations to generative layers. This study introduces a Pathological Clue-driven Representation Learning (PCRL) model to build cross-modal representations based on pathological clues and naturally adapt them for accurate report generation. Specifically, we construct pathological clues from perspectives of segmented regions, pathological entities, and report themes, to fully grasp visual pathological patterns and learn cross-modal feature representations. To adapt the representations for the text generation task, we bridge the gap between representation learning and report generation by using a unified large language model (LLM) with task-tailored instructions. These crafted instructions enable the LLM to be flexibly fine-tuned across tasks and smoothly transfer the semantic representation for report generation. Experiments demonstrate that our method outperforms previous methods and achieves SoTA performance. Our code is available at "https://github.com/Chauncey-Jheng/PCRL-MRG".
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 29 Sep 2024.
Builds on
31 of 49 references.
References · 31
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Segment Anythingcited by 8,234
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
- 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
- 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
- 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
- 2014Show and Tell: A Neural Image Caption Generatorcited by 879
- 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
19 more
- 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
- 2024The Platonic Representation Hypothesiscited by 382
- 2017On the Automatic Generation of Medical Imaging Reportscited by 216
- 2022Cross-modal Memory Networks for Radiology Report Generationcited by 185
- 2020When Radiology Report Generation Meets Knowledge Graphcited by 145
- 2018Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generationcited by 141
- 2018TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rayscited by 139
- 2023Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generationcited by 118
- 2016A Hierarchical Approach for Generating Descriptive Image Paragraphscited by 104
- 2019Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generationcited by 102
- 2023PromptMRG: Diagnosis-Driven Prompts for Medical Report Generationcited by 95
- 2022Competence-based Multimodal Curriculum Learning for Medical Report Generationcited by 76
- 2021Contrastive Attention for Automatic Chest X-ray Report Generationcited by 71
- 2021Knowledge Matters: Radiology Report Generation with General and Specific Knowledgecited by 68
- 2022AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generationcited by 65
- 2020Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reportscited by 58
- 2021Weakly Supervised Contrastive Learning for Chest X-Ray Report Generationcited by 41
- 2024Dia-LLaMA: Towards Large Language Model-driven CT Report Generationcited by 32
- 2018Development and Validation of Deep Learning Algorithms for Detection of Critical Findings in Head CT Scanscited by 24
Built on by
Citing papers per year: 2025: 2.
2 papers.