Cyclic Vision-Language Manipulator: Towards Reliable and Fine-Grained Image Interpretation for Automated Report Generation
Not on the map. Why?
Abstract (arXiv)
Despite significant advancements in automated report generation, the opaqueness of text interpretability continues to cast doubt on the reliability of the content produced. This paper introduces a novel approach to identify specific image features in X-ray images that influence the outputs of report generation models. Specifically, we propose Cyclic Vision-Language Manipulator CVLM, a module to generate a manipulated X-ray from an original X-ray and its report from a designated report generator. The essence of CVLM is that cycling manipulated X-rays to the report generator produces altered reports aligned with the alterations pre-injected into the reports for X-ray generation, achieving the term "cyclic manipulation". This process allows direct comparison between original and manipulated X-rays, clarifying the critical image features driving changes in reports and enabling model users to assess the reliability of the generated texts. Empirical evaluations demonstrate that CVLM can identify more precise and reliable features compared to existing explanation methods, significantly enhancing the transparency and applicability of AI-generated reports.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 32 of its 45 references that are in the map. First posted on arXiv 8 Nov 2024.
Builds on
32 of 45 references.
References · 32
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
- 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
- 2017Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPRcited by 1,086
- 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
- 2021DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulationcited by 640
- 2019MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographscited by 590
- 2020Generating Radiology Reports via Memory-driven Transformercited by 363
- 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
- 2022Cross-modal Memory Networks for Radiology Report Generationcited by 183
20 more
- 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 139
- 2019Clinically Accurate Chest X-Ray Report Generationcited by 138
- 2021Explaining in Style: Training a GAN to explain a classifier in StyleSpacecited by 122
- 2023METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokenscited by 98
- 2021CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directionscited by 78
- 2023LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generationcited by 57
- 2021Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanationscited by 56
- 2020GANterfactual - Counterfactual Explanations for Medical Non-Experts using Generative Adversarial Learningcited by 50
- 2024RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysiscited by 41
- 2021Using StyleGAN for Visual Interpretability of Deep Learning Models on Medical Imagescited by 35
- 2023DeltaEdit: Exploring Text-free Training for Text-Driven Image Manipulationcited by 27
- 2022CheXplaining in Style: Counterfactual Explanations for Chest X-rays using StyleGANcited by 25
- 2022CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generationcited by 21
- 2024ChEX: Interactive Localization and Region Description in Chest X-rayscited by 18
- 2023ACAT: Adversarial Counterfactual Attention for Classification and Detection in Medical Imagingcited by 9
- 2023Beyond Known Reality: Exploiting Counterfactual Explanations for Medical Researchcited by 9
- 2023PIE: Simulating Disease Progression via Progressive Image Editingcited by 6
- 2024DiffExplainer: Unveiling Black Box Models Via Counterfactual Generationcited by 5
- 2019Justifying Diagnosis Decisions by Deep Neural Networkscited by 3
- 2024Decoding Decision Reasoning: A Counterfactual-Powered Model for Knowledge Discoverycited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.