mlsanityData to 7 Oct 2026

LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention

arXiv 2010.01057 ↗First posted 2 Oct 2020cs.CL cs.LGcited by 326

Not on the map. Why?

Abstract (arXiv)

Entity representations are useful in natural language tasks involving entities. In this paper, we propose new pretrained contextualized representations of words and entities based on the bidirectional transformer. The proposed model treats words and entities in a given text as independent tokens, and outputs contextualized representations of them. Our model is trained using a new pretraining task based on the masked language model of BERT. The task involves predicting randomly masked words and entities in a large entity-annotated corpus retrieved from Wikipedia. We also propose an entity-aware self-attention mechanism that is an extension of the self-attention mechanism of the transformer, and considers the types of tokens (words or entities) when computing attention scores. The proposed model achieves impressive empirical performance on a wide range of entity-related tasks. In particular, it obtains state-of-the-art results on five well-known datasets: Open Entity (entity typing), TACRED (relation classification), CoNLL-2003 (named entity recognition), ReCoRD (cloze-style question answering), and SQuAD 1.1 (extractive question answering). Our source code and pretrained representations are available at https://github.com/studio-ousia/luke.

Key facts. Cited by 326 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 61; 2022: 100; 2023: 89; 2024: 47; 2025: 17; 2026: 12.

326 papers.

Sort:

By area:

  1. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  2. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  3. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  4. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  5. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 235
  6. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  7. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  8. 2022A Review on Language Models as Knowledge Basescited by 156
  9. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  10. 2021Template-Based Named Entity Recognition Using BARTcited by 153
  11. 2021A Unified Generative Framework for Various NER Subtaskscited by 141
  12. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
38 more
  1. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  2. 2023Detecting and Grounding Multi-Modal Media Manipulationcited by 98
  3. 2021Unified Named Entity Recognition as Word-Word Relation Classificationcited by 89
  4. 2021An Improved Baseline for Sentence-level Relation Extractioncited by 80
  5. 2021Improving Named Entity Recognition by External Context Retrieving and Cooperative Learningcited by 76
  6. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  7. 2021Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extractioncited by 73
  8. 2022OntoProtein: Protein Pretraining With Gene Ontology Embeddingcited by 70
  9. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  10. 2023Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Taskscited by 59
  11. 2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Taskscited by 58
  12. 2022UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Trainingcited by 57
  13. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  14. 2021SpanNER: Named Entity Re-/Recognition as Span Predictioncited by 55
  15. 2023Detecting and Grounding Multi-Modal Media Manipulation and Beyondcited by 55
  16. 2021ExplainaBoard: An Explainable Leaderboard for NLPcited by 50
  17. 2021Packed Levitated Marker for Entity and Relation Extractioncited by 49
  18. 2021Named Entity Recognition and Classification on Historical Documents: A Surveycited by 47
  19. 2023Reducing Privacy Risks in Online Self-Disclosures with Language Modelscited by 47
  20. 2023Towards Reliable Misinformation Mitigation: Generalization, Uncertainty, and GPT-4cited by 45
  21. 2022T-NER: An All-Round Python Library for Transformer-based Named Entity Recognitioncited by 37
  22. 2023GeoLM: Empowering Language Models for Geospatially Grounded Language Understandingcited by 37
  23. 2021Learning Rich Representation of Keyphrases from Textcited by 36
  24. 2021Learning from Noisy Labels for Entity-Centric Information Extractioncited by 34
  25. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  26. 2022UCTopic: Unsupervised Contrastive Learning for Phrase Representations and Topic Miningcited by 33
  27. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  28. 2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
  29. 2021Relation Classification with Entity Type Restrictioncited by 31
  30. 2022Summarization as Indirect Supervision for Relation Extractioncited by 31
  31. 2021CodeQA: A Question Answering Dataset for Source Code Comprehensioncited by 30
  32. 2022Should We Rely on Entity Mentions for Relation Extraction? Debiasing Relation Extraction with Counterfactual Analysiscited by 30
  33. 2023How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?cited by 30
  34. 2023Revisiting Large Language Models as Zero-shot Relation Extractorscited by 29
  35. 2022Naamapadam: A Large-Scale Named Entity Annotated Data for Indic Languagescited by 28
  36. 2021LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Promptingcited by 27
  37. 2022Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shiftscited by 27
  38. 2022Named Entity Recognition in Indian court judgmentscited by 26

50 most cited of 326 All 326 (JSON)