mlsanityData to 7 Oct 2026

FigEx2: Visual-Conditioned Panel Detection and Captioning for Scientific Compound Figures

arXiv 2601.08026 ↗First posted 12 Jan 2026cs.CV cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Scientific compound figures combine multiple labeled panels into a single image, and downstream pretraining and retrieval require panel-aligned visual-text pairs. However, in a PubMed Central (PMC)-scale crawl of 346,567 compound figures, 16.3% have no caption and are discarded by existing caption-decomposition pipelines. We propose FigEx2, a visual-conditioned framework that takes only a compound figure as input and jointly produces labeled panel boxes and panel-wise captions. FigEx2 introduces an Entity-Attention Kullback-Leibler (KL) regularizer that aligns the detector's cross-attention with scientific entities annotated for each panel, providing a stable conditioning signal that also improves localization, and applies Group Relative Policy Optimization (GRPO) with a panel-level Entity-F1 reward to optimize scientific faithfulness. We curate BioSci-Fig-Cap for in-domain supervision and contribute physics and chemistry test suites for cross-disciplinary evaluation. FigEx2 achieves 0.751 [email protected]:0.95 on BioSci-Fig-Cap, and outperforms Qwen3-VL-8B by 6.80 Entity-F1 on MedICaT for captioning. It also transfers zero-shot to out-of-distribution domains. The source code is available at https://github.com/Huang-AI4Medicine-Lab/FigEx2.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 57 references that are in the map. First posted on arXiv 12 Jan 2026.

Builds on

31 of 57 references.

References · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  5. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  6. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  7. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  8. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  9. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2022DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRcited by 569
  12. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
19 more
  1. 2016Self-critical Sequence Training for Image Captioningcited by 544
  2. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  3. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  4. 2021You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectioncited by 170
  5. 2022Improving the Factual Correctness of Radiology Report Generation with Semantic Rewardscited by 82
  6. 2020MedICaT: A Dataset of Medical Images, Captions, and Textual Referencescited by 71
  7. 2022Fine-grained Image Captioning with CLIP Rewardcited by 69
  8. 2022BERN2: an advanced neural biomedical named entity recognition and normalization toolcited by 42
  9. 2017Deep Reinforcement Learning-based Image Captioning with Embedding Rewardcited by 39
  10. 2023Lenna: Language Enhanced Reasoning Detection Assistantcited by 29
  11. 2022DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Groundingcited by 24
  12. 2024VLRM: Vision-Language Models act as Reward Models for Image Captioningcited by 18
  13. 2016Viziometrics: Analyzing Visual Information in the Scientific Literaturecited by 14
  14. 2024SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluationcited by 11
  15. 2024X-ray Made Simple: Lay Radiology Report Generation and Robust Evaluationcited by 9
  16. 2025Group Relative Policy Optimization for Image Captioningcited by 9
  17. 2021A Two-stage Framework for Compound Figure Separationcited by 8
  18. 2024SimLTD: Simple Supervised and Semi-Supervised Long-Tailed Object Detectioncited by 4
  19. 2025Zeus: Zero-shot LLM Instruction for Union Segmentation in Multimodal Medical Imagingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025From Panel to Pixel: Zoom-In Vision-Language Pretraining from Biomedical Scientific Literature
  2. 2025RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioning
  3. 2024MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding
  4. 2026FigAct: Turning Scientific Figures into Active Canvases for Explanation
  5. 2026S1-MMAlign: A Large-Scale, Multi-Disciplinary Dataset for Scientific Figure-Text Understanding
  6. 2023FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedback
  7. 2026Edit2TikZ: A Comprehensive and Challenging Benchmark for Scientific Figure Editing with TikZ
  8. 2025GTR-CoT: Graph Traversal as Visual Chain of Thought for Molecular Structure Recognition