mlsanityData to 7 Oct 2026

Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment

arXiv 2305.12260 ↗First posted 20 May 2023cs.CV cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Unpaired cross-lingual image captioning has long suffered from irrelevancy and disfluency issues, due to the inconsistencies of the semantic scene and syntax attributes during transfer. In this work, we propose to address the above problems by incorporating the scene graph (SG) structures and the syntactic constituency (SC) trees. Our captioner contains the semantic structure-guided image-to-pivot captioning and the syntactic structure-guided pivot-to-target translation, two of which are joined via pivot language. We then take the SG and SC structures as pivoting, performing cross-modal semantic structure alignment and cross-lingual syntactic structure alignment learning. We further introduce cross-lingual&cross-modal back-translation training to fully align the captioning and translation stages. Experiments on English-Chinese transfers show that our model shows great superiority in improving captioning relevancy and fluency.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 49 of its 91 references that are in the map. First posted on arXiv 20 May 2023.

Builds on

49 of 91 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  8. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  9. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  10. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  11. 2018Image Generation from Scene Graphscited by 324
  12. 2016Image Captioning with Semantic Attentioncited by 260
37 more
  1. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 254
  2. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  3. 2018An efficient framework for learning sentence representationscited by 197
  4. 2018Constituency Parsing with a Self-Attentive Encodercited by 196
  5. 2017Encoding Sentences with Graph Convolutional Networks for Semantic Role Labelingcited by 137
  6. 2016Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challengecited by 116
  7. 2014From Captions to Visual Concepts and Backcited by 109
  8. 2021Unified Named Entity Recognition as Word-Word Relation Classificationcited by 89
  9. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  10. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 79
  11. 2017AI Challenger : A Large-scale Dataset for Going Deeper in Image Understandingcited by 73
  12. 2020Cross-Lingual Semantic Role Labeling with High-Quality Translated Training Corpuscited by 49
  13. 2020Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Networkcited by 48
  14. 2023LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Modelcited by 43
  15. 2017Fluency-Guided Cross-Lingual Image Captioningcited by 33
  16. 2021Learn from Syntax: Improving Pair-wise Aspect and Opinion Terms Extractionwith Rich Syntactic Knowledgecited by 31
  17. 2017Stack-Captioning: Coarse-to-Fine Learning for Image Captioningcited by 27
  18. 2018Unpaired Image Captioning by Language Pivotingcited by 27
  19. 2022Neural Machine Translation with Phrase-Level Universal Visual Representationscited by 26
  20. 2020Improving Disfluency Detection by Self-Training a Self-Attentive Modelcited by 25
  21. 2021Mastering the Explicit Opinion-role Interaction: Syntax-aided Neural Transition System for Unified Opinion Role Labelingcited by 25
  22. 2020Retrofitting Structure-aware Transformer Language Model for End Taskscited by 24
  23. 2020Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivotingcited by 22
  24. 2021End-to-end Semantic Role Labeling with Neural Transition-based Modelcited by 22
  25. 2022Effective Token Graph Modeling using a Novel Labeling Strategy for Structured Sentiment Analysiscited by 22
  26. 2018Scene Graph Parsing as Dependency Parsingcited by 21
  27. 2023On the Robustness of Aspect-based Sentiment Analysis: Rethinking Model, Data, and Trainingcited by 20
  28. 2022Entity-centered Cross-document Relation Extractioncited by 18
  29. 2020Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLPcited by 15
  30. 2022Conversational Semantic Role Labeling with Predicate-Oriented Latent Graphcited by 14
  31. 2019Neural Constituency Parsing of Speech Transcriptscited by 10
  32. 2022Joint Alignment of Multi-Task Feature and Label Spaces for Emotion Cause Pair Extractioncited by 10
  33. 2019Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewardscited by 9
  34. 2020UNISON: Unpaired Cross-lingual Image Captioningcited by 8
  35. 2022Conversation Disentanglement with Bi-Level Contrastive Learningcited by 6
  36. 2020Learning Source Phrase Representations for Neural Machine Translationcited by 4
  37. 2022CSynGEC: Incorporating Constituent-based Syntax for Grammatical Error Correction with a Tailored GEC-Oriented Parsercited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 6; 2025: 2; 2026: 1.

15 papers.

Sort:

By area:

  1. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 151
  2. 2023LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generationcited by 92
  3. 2023Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labelingcited by 15
  4. 2023Revisiting Disentanglement and Fusion on Modality and Context in Conversational Multimodal Emotion Recognitioncited by 13
  5. 2025Learning 4D Panoptic Scene Graph Generation from Rich 2D Visual Scenecited by 6
  6. 2024NUS-Emo at SemEval-2024 Task 3: Instruction-Tuning LLM for Multimodal Emotion-Cause Analysis in Conversationscited by 5
  7. 2023Semi-Supervised Panoptic Narrative Groundingcited by 4
  8. 2023A Bi-directional Multi-hop Inference Model for Joint Dialog Sentiment Classification and Act Recognitioncited by 2
  9. 2023DialogRE^C+: An Extension of DialogRE to Investigate How Much Coreference Helps Relation Extraction in Dialogscited by 1
  10. 2024Revisiting Structured Sentiment Analysis as Latent Dependency Graph Parsingcited by 1
  11. 2024Synergistic Dual Spatial-aware Generation of Image-to-Text and Text-to-Imagecited by 1
  12. 2025Align Where the Words Look: Cross-Attention-Guided Patch Alignment with Contrastive and Transport Regularization for Bengali Captioningcited by 1
3 more
  1. 2024Modeling Unified Semantic Discourse Structure for High-quality Headline Generationcited by 0
  2. 2024Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysiscited by 0
  3. 2026MUNIChus: Multilingual News Image Captioning Benchmarkcited by 0