# Token Imbalance Adaptation for Radiology Report Generation

arXiv: https://arxiv.org/abs/2304.09185 · HTML: https://mlsanity.com/paper/2304.09185/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 18 Apr 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Neural Text Generation with Unlikelihood Training](https://mlsanity.com/paper/1908.04319/) (2019)
- [Generating Radiology Reports via Memory-driven Transformer](https://mlsanity.com/paper/2010.16056/) (2020)
- [CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT](https://mlsanity.com/paper/2004.09167/) (2020)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Cross-modal Memory Networks for Radiology Report Generation](https://mlsanity.com/paper/2204.13258/) (2022)
- [Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation](https://mlsanity.com/paper/2106.06963/) (2021)
- [An Actor-Critic Algorithm for Sequence Prediction](https://mlsanity.com/paper/1607.07086/) (2016)
- [Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation](https://mlsanity.com/paper/2010.10042/) (2020)
- [Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards](https://mlsanity.com/paper/2210.12186/) (2022)
- [Competence-based Multimodal Curriculum Learning for Medical Report Generation](https://mlsanity.com/paper/2206.14579/) (2022)
- [Contrastive Attention for Automatic Chest X-ray Report Generation](https://mlsanity.com/paper/2106.06965/) (2021)
- [Knowledge Matters: Radiology Report Generation with General and Specific Knowledge](https://mlsanity.com/paper/2112.15009/) (2021)
- [Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-Ray Reports](https://mlsanity.com/paper/2004.12274/) (2020)
- [Toward Diverse Text Generation with Inverse Reinforcement Learning](https://mlsanity.com/paper/1804.11258/) (2018)
- [Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings](https://mlsanity.com/paper/2109.03127/) (2021)
- [Automated Generation of Accurate & Fluent Medical X-ray Reports](https://mlsanity.com/paper/2108.12126/) (2021)
- [Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation](https://mlsanity.com/paper/2206.01988/) (2022)
- [Addressing Data Bias Problems for Chest X-ray Image Report Generation](https://mlsanity.com/paper/1908.02123/) (2019)
- [Teacher Forcing Recovers Reward Functions for Text Generation](https://mlsanity.com/paper/2210.08708/) (2022)
- [Modeling the Unigram Distribution](https://mlsanity.com/paper/2106.02289/) (2021)
- [Unsupervised Reinforcement Adaptation for Class-Imbalanced Text Classification](https://mlsanity.com/paper/2205.13139/) (2022)

## Built on by (most cited first, 10 of 10)

- [Automated Radiology Report Generation: A Review of Recent Advances](https://mlsanity.com/paper/2405.10842/) (2024)
- [CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset](https://mlsanity.com/paper/2410.00379/) (2024)
- [Time Matters: Examine Temporal Effects on Biomedical Language Models](https://mlsanity.com/paper/2407.17638/) (2024)
- [A Survey of Deep Learning-based Radiology Report Generation Using Multimodal Data](https://mlsanity.com/paper/2405.12833/) (2024)
- [A Disease-Aware Dual-Stage Framework for Chest X-ray Report Generation](https://mlsanity.com/paper/2511.12259/) (2025)
- [Examining Imbalance Effects on Performance and Demographic Fairness of Clinical Language Models](https://mlsanity.com/paper/2412.17803/) (2024)
- [EMRRG: Efficient Fine-Tuning Pre-trained X-ray Mamba Networks for Radiology Report Generation](https://mlsanity.com/paper/2510.16776/) (2025)
- [OraPO: Oracle-educated Reinforcement Learning for Data-efficient and Factual Radiology Report Generation](https://mlsanity.com/paper/2509.18600/) (2025)
- [Seeing What Matters: Lesion-Aware High-Resolution Patch Discovery and Fusion for Chest X-ray Report Generation](https://mlsanity.com/paper/2607.06909/) (2026)
- [Pre-training, Reasoning, Benchmarking: X-ray Report Generation on CheXpert Plus Dataset](https://mlsanity.com/paper/2610.08813/) (2026)

## Abstract (arXiv)

Imbalanced token distributions naturally exist in text documents, leading neural language models to overfit on frequent tokens. The token imbalance may dampen the robustness of radiology report generators, as complex medical terms appear less frequently but reflect more medical information. In this study, we demonstrate how current state-of-the-art models fail to generate infrequent tokens on two standard benchmark datasets (IU X-RAY and MIMIC-CXR) of radiology report generation. % However, no prior study has proposed methods to adapt infrequent tokens for text generators feeding with medical images. To solve the challenge, we propose the Token Imbalance Adapter (textitTIMER), aiming to improve generation robustness on infrequent tokens. The model automatically leverages token imbalance by an unlikelihood loss and dynamically optimizes generation processes to augment infrequent tokens. We compare our approach with multiple state-of-the-art methods on the two benchmarks. Experiments demonstrate the effectiveness of our approach in enhancing model robustness overall and infrequent tokens. Our ablation analysis shows that our reinforcement learning method has a major effect in adapting token imbalance for radiology report generation.