mlsanityData to 14 Oct 2026

METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens

arXiv 2304.02211 ↗First posted 5 Apr 2023cs.CVcited by 101

Not on the map. Why?

Abstract (arXiv)

In clinical scenarios, multi-specialist consultation could significantly benefit the diagnosis, especially for intricate cases. This inspires us to explore a "multi-expert joint diagnosis" mechanism to upgrade the existing "single expert" framework commonly seen in the current literature. To this end, we propose METransformer, a method to realize this idea with a transformer-based backbone. The key design of our method is the introduction of multiple learnable "expert" tokens into both the transformer encoder and decoder. In the encoder, each expert token interacts with both vision tokens and other expert tokens to learn to attend different image regions for image representation. These expert tokens are encouraged to capture complementary information by an orthogonal loss that minimizes their overlap. In the decoder, each attended expert token guides the cross-attention between input words and visual tokens, thus influencing the generated report. A metrics-based expert voting strategy is further developed to generate the final report. By the multi-experts concept, our model enjoys the merits of an ensemble-based approach but through a manner that is computationally more efficient and supports more sophisticated interactions among experts. Experimental results demonstrate the promising performance of our proposed model on two widely used benchmarks. Last but not least, the framework-level innovation makes our work ready to incorporate advances on existing "single-expert" models to further improve its performance.

Key facts. Cited by 101 papers in the map (data through 14 Oct 2026). Builds on 34 of its 51 references that are in the map. First posted on arXiv 5 Apr 2023.

Builds on

34 of 51 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2016Layer Normalizationcited by 5,669
  6. 2017Squeeze-and-Excitation Networkscited by 4,789
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  9. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,423
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
22 more
  1. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  2. 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 597
  3. 2020Generating Radiology Reports via Memory-driven Transformercited by 368
  4. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  5. 2016Image Captioning with Semantic Attentioncited by 260
  6. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 232
  7. 2015Bilinear CNNs for Fine-grained Visual Recognitioncited by 228
  8. 2017On the Automatic Generation of Medical Imaging Reportscited by 216
  9. 2020X-Linear Attention Networks for Image Captioningcited by 198
  10. 2022Cross-modal Memory Networks for Radiology Report Generationcited by 185
  11. 2021Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generationcited by 162
  12. 2020When Radiology Report Generation Meets Knowledge Graphcited by 145
  13. 2018Hybrid Retrieval-Generation Reinforced Agent for Medical Image Report Generationcited by 141
  14. 2019Knowledge-driven Encode, Retrieve, Paraphrase for Medical Image Report Generationcited by 102
  15. 2021Radiology Report Generation with a Learned Knowledge Base and Multi-modal Alignmentcited by 73
  16. 2022AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generationcited by 65
  17. 2019Automatic Radiology Report Generation based on Multi-view Image Fusion and Medical Concept Enrichmentcited by 60
  18. 2021Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generationcited by 47
  19. 2022A Medical Semantic-Assisted Transformer for Radiographic Report Generationcited by 44
  20. 2020More Grounded Image Captioning by Distilling Image-Text Matching Modelcited by 43
  21. 2020Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generationcited by 35
  22. 2021X-modaler: A Versatile and High-performance Codebase for Cross-modal Analyticscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 36; 2025: 29; 2026: 27.

101 papers.

Sort:

By area:

  1. 2024MAIRA-2: Grounded Radiology Report Generationcited by 145
  2. 2023R2GenGPT: Radiology Report Generation with Frozen LLMscited by 114
  3. 2023PromptMRG: Diagnosis-Driven Prompts for Medical Report Generationcited by 95
  4. 2024Automated Radiology Report Generation: A Review of Recent Advancescited by 43
  5. 2025Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generationcited by 42
  6. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  7. 2024CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Datasetcited by 35
  8. 2024HERGen: Elevating Radiology Report Generation with Longitudinal Datacited by 33
  9. 2024Libra: Leveraging Temporal Images for Biomedical Radiology Analysiscited by 29
  10. 2023RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoningcited by 22
  11. 2023Complex Organ Mask Guided Radiology Report Generationcited by 22
  12. 2024ChEX: Interactive Localization and Region Description in Chest X-rayscited by 19
38 more
  1. 2024Self-eXplainable AI for Medical Image Analysis: A Survey and New Outlookscited by 18
  2. 2024M4CXR: Exploring Multi-task Potentials of Multi-modal Large Language Models for Chest X-ray Interpretationcited by 17
  3. 2024Radiology Report Generation via Multi-objective Preference Optimizationcited by 16
  4. 2024Structural Entities Extraction and Patient Indications Incorporation for Chest X-ray Report Generationcited by 15
  5. 2023A Systematic Review of Deep Learning-based Research on Radiology Report Generationcited by 14
  6. 2025Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generationcited by 13
  7. 2025RADAR: Enhancing Radiology Report Generation with Supplementary Knowledge Injectioncited by 13
  8. 2023Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generationcited by 12
  9. 2024R2GenCSR: Mining Contextual and Residual Information for LLMs-based Radiology Report Generationcited by 12
  10. 2024Factual Serialization Enhancement: A Key Innovation for Chest X-ray Report Generationcited by 11
  11. 2024A Survey of Medical Vision-and-Language Applications and Their Techniquescited by 11
  12. 2024CheX-GPT: Harnessing Large Language Models for Enhanced Chest X-ray Report Labelingcited by 10
  13. 2024HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generationcited by 9
  14. 2025DART: Disease-aware Image-Text Alignment and Self-correcting Re-alignment for Trustworthy Radiology Report Generationcited by 9
  15. 2025PriorRG: Prior-Guided Contrastive Pre-training and Coarse-to-Fine Decoding for Chest X-ray Report Generationcited by 9
  16. 2024Dragonfly: Multi-Resolution Zoom-In Encoding Enhances Vision-Language Modelscited by 8
  17. 2024Contrastive Learning with Counterfactual Explanations for Radiology Report Generationcited by 8
  18. 2025DDaTR: Dynamic Difference-aware Temporal Residual Network for Longitudinal Radiology Report Generationcited by 8
  19. 2024Pre-training on High Definition X-ray Images: An Experimental Studycited by 7
  20. 2024KARGEN: Knowledge-enhanced Automated Radiology Report Generation Using Large Language Modelscited by 7
  21. 2025A Chain of Diagnosis Framework for Accurate and Explainable Radiology Report Generationcited by 7
  22. 2024Prompt-Guided Generation of Structured Chest X-Ray Report Using a Pre-trained LLMcited by 6
  23. 2024A Survey of Deep Learning-based Radiology Report Generation Using Multimodal Datacited by 6
  24. 2024VoxelPrompt: A Vision Agent for End-to-End Medical Image Analysiscited by 6
  25. 2025S-RRG-Bench: Structured Radiology Report Generation with Fine-Grained Evaluation Frameworkcited by 6
  26. 2024Topicwise Separable Sentence Retrieval for Medical Report Generationcited by 5
  27. 2025CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationalecited by 5
  28. 2025A Disease-Aware Dual-Stage Framework for Chest X-ray Report Generationcited by 5
  29. 2025Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generationcited by 5
  30. 2026XrayClaw: Cooperative-Competitive Multi-Agent Alignment for Trustworthy Chest X-ray Diagnosiscited by 5
  31. 2023Act Like a Radiologist: Radiology Report Generation across Anatomical Regionscited by 4
  32. 2024EVOKE: Elevating Chest X-ray Report Generation via Multi-View Contrastive Learning and Patient-Specific Knowledgecited by 4
  33. 2025Walking the Tightrope: Disentangling Beneficial and Detrimental Drifts in Non-Stationary Custom-Tuningcited by 4
  34. 2025LLM-driven Medical Report Generation via Communication-efficient Heterogeneous Federated Learningcited by 4
  35. 2025R2GenKG: Hierarchical Multi-modal Knowledge Graph for LLM-based Radiology Report Generationcited by 4
  36. 2026MARCH: Multi-Agent Radiology Clinical Hierarchy for CT Report Generationcited by 4
  37. 2024TiBiX: Leveraging Temporal Information for Bidirectional X-ray and Report Generationcited by 3
  38. 2024Image-aware Evaluation of Generated Medical Reportscited by 3

50 most cited of 101 All 101 (JSON)