mlsanityData to 14 Oct 2026

Generative Multimodal Entity Linking

arXiv 2306.12725 ↗First posted 22 Jun 2023cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Multimodal Entity Linking (MEL) is the task of mapping mentions with multimodal contexts to the referent entities from a knowledge base. Existing MEL methods mainly focus on designing complex multimodal interaction mechanisms and require fine-tuning all model parameters, which can be prohibitively costly and difficult to scale in the era of Large Language Models (LLMs). In this work, we propose GEMEL, a Generative Multimodal Entity Linking framework based on LLMs, which directly generates target entity names. We keep the vision and language model frozen and only train a feature mapper to enable cross-modality interactions. To adapt LLMs to the MEL task, we leverage the in-context learning capability of LLMs by retrieving multimodal instances as demonstrations. Extensive experiments show that, with only ~0.3% of the model parameters fine-tuned, GEMEL achieves state-of-the-art results on two well-established MEL datasets (7.7% accuracy gains on WikiDiverse and 8.8% accuracy gains on WikiMEL). The performance gain stems from mitigating the popularity bias of LLM predictions and disambiguating less common entities effectively. Further analysis verifies the generality and scalability of GEMEL. Our framework is compatible with any off-the-shelf language model, paving the way towards an efficient and general solution for utilizing LLMs in the MEL task. Our code is available at https://github.com/HITsz-TMG/GEMEL.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 34 of its 58 references that are in the map. First posted on arXiv 22 Jun 2023.

Builds on

34 of 58 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2023GPT-4 Technical Reportcited by 18,396
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
22 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  3. 2022Emergent Abilities of Large Language Modelscited by 2,512
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  9. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  10. 2016Neural Architectures for Named Entity Recognitioncited by 550
  11. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  12. 2020Autoregressive Entity Retrievalcited by 334
  13. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  14. 2022Linearly Mapping from Image to Text Spacecited by 142
  15. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  16. 2018Question Answering by Reasoning Across Documents with Graph Convolutional Networkscited by 56
  17. 2023ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfercited by 50
  18. 2022WikiDiverse: A Multimodal Entity Linking Dataset with Diversified Contextual Topics and Entity Typescited by 34
  19. 2021Evaluating Entity Disambiguation and the Role of Popularity in Retrieval-Based NLPcited by 31
  20. 2021Multimodal Entity Linking for Tweetscited by 28
  21. 2022Improving Candidate Retrieval with Entity Profile Generation for Wikidata Entity Linkingcited by 10
  22. 2021Towards Consistent Document-level Entity Linking: Joint Models for Entity Linking and Coreference Resolutioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 2.

12 papers.

  1. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 53
  2. 2024MKGL: Mastery of a Three-Word Languagecited by 17
  3. 2025Digital Twin Generation from Visual Data: A Surveycited by 7
  4. 2026LELA: an LLM-based Entity Linking Approach with Zero-Shot Domain Adaptationcited by 6
  5. 2024Web-Scale Visual Entity Recognition: An LLM-Driven Data Approachcited by 5
  6. 2024Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Modelscited by 4
  7. 2025DeepMEL: A Multi-Agent Collaboration Framework for Multimodal Entity Linkingcited by 3
  8. 2024Advancing Grounded Multimodal Named Entity Recognition via LLM-Based Reformulation and Box-Based Segmentationcited by 1
  9. 2025Mutual Understanding between People and Systems via Neurosymbolic AI and Knowledge Graphscited by 1
  10. 2025MERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linkingcited by 1
  11. 2025PGMEL: Policy Gradient-based Generative Adversarial Network for Multimodal Entity Linkingcited by 0
  12. 2026KARAMEL: Knowledge-Aware Ranking for Multimodal Entity Linking in the Arts via Heterogeneous Graphscited by 0