mlsanityData to 7 Oct 2026

Exploring the Implicit Semantic Ability of Multimodal Large Language Models: A Pilot Study on Entity Set Expansion

arXiv 2501.00330 ↗First posted 31 Dec 2024cs.CL cs.AI cs.IRcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The rapid development of multimodal large language models (MLLMs) has brought significant improvements to a wide range of tasks in real-world applications. However, LLMs still exhibit certain limitations in extracting implicit semantic information. In this paper, we apply MLLMs to the Multi-modal Entity Set Expansion (MESE) task, which aims to expand a handful of seed entities with new entities belonging to the same semantic class, and multi-modal information is provided with each entity. We explore the capabilities of MLLMs to understand implicit semantic information at the entity-level granularity through the MESE task, introducing a listwise ranking method LUSAR that maps local scores to global rankings. Our LUSAR demonstrates significant improvements in MLLM's performance on the MESE task, marking the first use of generative MLLM for ESE tasks and extending the applicability of listwise ranking.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 33 references that are in the map. First posted on arXiv 31 Dec 2024.

Builds on

30 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  7. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  8. 2021Are we ready for a new paradigm shift? A Survey on Visual Deep MLPcited by 63
  9. 2024LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewingcited by 62
  10. 2022The Past Mistake is the Future Wisdom: Error-driven Contrastive Probability Optimization for Chinese Spell Checkingcited by 56
  11. 2023On the (In)Effectiveness of Large Language Models for Chinese Text Correctioncited by 53
  12. 2023EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commercecited by 48
17 more
  1. 2023LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzlescited by 41
  2. 2022Learning from the Dictionary: Heterogeneous Knowledge Guided Fine-tuning for Chinese Spell Checkingcited by 31
  3. 2022Contrastive Learning with Hard Negative Entities for Entity Set Expansioncited by 26
  4. 2019SetExpan: Corpus-Based Set Expansion via Context Feature Selection and Rank Ensemblecited by 25
  5. 2020Empower Entity Set Expansion via Language Model Probingcited by 25
  6. 2022Linguistic Rules-Based Corpus Generation for Native Chinese Grammatical Error Correctioncited by 25
  7. 2024When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Modelscited by 22
  8. 2024Evaluating and Aligning CodeLLMs on Human Preferencecited by 22
  9. 2022Automatic Context Pattern Generation for Entity Set Expansioncited by 21
  10. 2023Towards Real-World Writing Assistance: A Chinese Character Checking Benchmark with Faked and Misspelled Characterscited by 21
  11. 2024ExecRepoBench: Multi-level Executable Code Completion Evaluationcited by 19
  12. 2023Correct Like Humans: Progressive Learning Framework for Chinese Text Error Correctioncited by 13
  13. 2023MESED: A Multi-modal Entity Set Expansion Dataset with Fine-grained Semantic Classes and Hard Negative Entitiescited by 13
  14. 2022Active Relation Discovery: Towards General and Label-aware Open Relation Extractioncited by 10
  15. 2024Rethinking the Roles of Large Language Models in Chinese Grammatical Error Correctioncited by 9
  16. 2023CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correctioncited by 5
  17. 2023From Retrieval to Generation: Efficient and Effective Entity Set Expansioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  2. 2026TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoningcited by 3
  3. 2025Teaching According to Talents! Instruction Tuning LLMs with Competence-Aware Curriculum Learningcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editing
  2. 2024UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models
  3. 2024Personalized Multimodal Large Language Models: A Survey
  4. 2025MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models
  5. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers
  6. 2024Multilingual Large Language Models: A Systematic Survey
  7. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  8. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective