K-ON: Stacking Knowledge On the Head Layer of Large Language Model
Not on the map. Why?
Abstract (arXiv)
Recent advancements in large language models (LLMs) have significantly improved various natural language processing (NLP) tasks. Typically, LLMs are trained to predict the next token, aligning well with many NLP tasks. However, in knowledge graph (KG) scenarios, entities are the fundamental units and identifying an entity requires at least several tokens. This leads to a granularity mismatch between KGs and natural languages. To address this issue, we propose K-ON, which integrates KG knowledge into the LLM by employing multiple head layers for next k-step prediction. K-ON can not only generate entity-level results in one step, but also enables contrastive loss against entities, which is the most powerful tool in KG representation learning. Experimental results show that K-ON outperforms state-of-the-art methods that incorporate text and even the other modalities.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 10 Feb 2025.
Builds on
22 of 40 references.
References · 22
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
- 2014Embedding Entities and Relations for Learning and Inference in Knowledge Basescited by 929
- 2019RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Spacecited by 893
- 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
- 2017Convolutional 2D Knowledge Graph Embeddingscited by 729
- 2022A Survey on In-context Learningcited by 717
- 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
- 2019Composition-based Multi-Relational Graph Convolutional Networkscited by 395
- 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
- 2019KG-BERT: BERT for Knowledge Graph Completioncited by 323
- 2023The Learnability of In-Context Learningcited by 126
10 more
- 2019MMKG: Multi-Modal Knowledge Graphscited by 104
- 2019Learning to Exploit Long-term Relational Dependencies in Knowledge Graphscited by 83
- 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
- 2023Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofittingcited by 70
- 2016Image-embodied Knowledge Representation Learningcited by 68
- 2020HittER: Hierarchical Transformers for Knowledge Graph Embeddingscited by 54
- 2022MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybridcited by 29
- 2024NativE: Multi-modal Knowledge Graph Completion in the Wildcited by 23
- 2022KGLM: Integrating Knowledge Graph Structure in Language Models for Link Predictioncited by 15
- 2022Unleashing the Power of Transformer for Graphscited by 6
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.