mlsanityData to 14 Oct 2026

Benchmarking Diverse-Modal Entity Linking with Generative Models

arXiv 2305.17337 ↗First posted 27 May 2023cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Entities can be expressed in diverse formats, such as texts, images, or column names and cell values in tables. While existing entity linking (EL) models work well on per modality configuration, such as text-only EL, visual grounding, or schema linking, it is more challenging to design a unified model for diverse modality configurations. To bring various modality configurations together, we constructed a benchmark for diverse-modal EL (DMEL) from existing EL datasets, covering all three modalities including text, image, and table. To approach the DMEL task, we proposed a generative diverse-modal model (GDMM) following a multimodal-encoder-decoder paradigm. Pre-training Model with rich corpora builds a solid foundation for DMEL without storing the entire KB for inference. Fine-tuning GDMM builds a stronger DMEL baseline, outperforming state-of-the-art task-specific EL models by 8.51 F1 score on average. Additionally, extensive error analyses are conducted to highlight the challenges of DMEL, facilitating future research on this task.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 56 of its 80 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

56 of 80 references.

References · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  10. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  12. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
38 more
  1. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  2. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 687
  3. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  5. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  6. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
  7. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  8. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  9. 2019RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parserscited by 424
  10. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  11. 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 408
  12. 2020Autoregressive Entity Retrievalcited by 334
  13. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  14. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  15. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 279
  16. 2021DocFormer: End-to-End Transformer for Document Understandingcited by 256
  17. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 235
  18. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  19. 2019Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representationcited by 210
  20. 2022Visual Attention Networkcited by 210
  21. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  22. 2018Improving Text-to-SQL Evaluation Methodologycited by 135
  23. 2021Towards Robustness of Text-to-SQL Models against Synonym Substitutioncited by 110
  24. 2021Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalizationcited by 92
  25. 2021LaTr: Layout-Aware Transformer for Scene-Text VQAcited by 72
  26. 2020On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queriescited by 60
  27. 2018Improving Entity Linking by Modeling Latent Relations between Mentionscited by 53
  28. 2020Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linkingcited by 49
  29. 2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
  30. 2021EntQA: Entity Linking as Question Answeringcited by 34
  31. 2022WikiDiverse: A Multimodal Entity Linking Dataset with Diversified Contextual Topics and Entity Typescited by 34
  32. 2022Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsingcited by 30
  33. 2021Multimodal Entity Linking for Tweetscited by 28
  34. 2021Awakening Latent Grounding from Pretrained Language Models for Semantic Parsingcited by 28
  35. 2019Joint Learning of Named Entity Recognition and Entity Linkingcited by 26
  36. 2021Highly Parallel Autoregressive Entity Linking with Discriminative Correctioncited by 21
  37. 2022NLIP: Noise-robust Language-Image Pre-trainingcited by 20
  38. 2022YORO -- Lightweight End to End Visual Groundingcited by 17

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 2.

5 papers.

  1. 2024MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editingcited by 25
  2. 2024OneNet: A Fine-Tuning Free Framework for Few-Shot Entity Linking via Large Language Model Promptingcited by 9
  3. 2024UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Modelscited by 5
  4. 2025Contextual Augmentation for Entity Linking using Large Language Modelscited by 5
  5. 2025KGMEL: Knowledge Graph-Enhanced Multimodal Entity Linkingcited by 4