Do Models Learn the Directionality of Relations? A New Evaluation: Relation Direction Recognition
Not on the map. Why?
Abstract (arXiv)
Deep neural networks such as BERT have made great progress in relation classification. Although they can achieve good performance, it is still a question of concern whether these models recognize the directionality of relations, especially when they may lack interpretability. To explore the question, a novel evaluation task, called Relation Direction Recognition (RDR), is proposed to explore whether models learn the directionality of relations. Three metrics for RDR are introduced to measure the degree to which models recognize the directionality of relations. Several state-of-the-art models are evaluated on RDR. Experimental results on a real-world dataset indicate that there are clear gaps among them in recognizing the directionality of relations, even though these models obtain similar performance in the traditional metric (e.g. Macro-F1). Finally, some suggestions are discussed to enhance models to recognize the directionality of relations from the perspective of model design or training.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 28 of its 46 references that are in the map. First posted on arXiv 19 May 2021.
Builds on
28 of 46 references.
References · 28
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
- 2019Is Attention Interpretable?cited by 387
- 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
- 2019Knowledge Enhanced Contextual Word Representationscited by 294
- 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
- 2019Matching the Blanks: Distributional Similarity for Relation Learningcited by 271
- 2016End-to-End Relation Extraction using LSTMs on Sequences and Tree Structurescited by 161
- 2018Graph Convolution over Pruned Dependency Trees Improves Relation Extractioncited by 146
- 2019Enriching Pre-trained Language Model with Entity Information for Relation Classificationcited by 82
16 more
- 2019Attention Guided Graph Convolutional Networks for Relation Extractioncited by 82
- 2019SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominalscited by 63
- 2015Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Pathcited by 35
- 2015Classifying Relations by Ranking with Convolutional Neural Networkscited by 33
- 2021Relation Classification with Entity Type Restrictioncited by 31
- 2016Improved Relation Classification by Deep Recurrent Neural Networks with Data Augmentationcited by 26
- 2018Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inferencecited by 26
- 2017Hierarchical Embeddings for Hypernymy Detection and Directionalitycited by 19
- 2019Extracting Multiple-Relations in One-Pass with Pre-Trained Transformerscited by 19
- 2019TransGCN:Coupling Transformation Assumptions with Graph Convolutional Networks for Link Predictioncited by 17
- 2015Semantic Relation Classification via Convolutional Neural Networks with Simple Negative Samplingcited by 12
- 2020Relation Classification as Two-way Span-Predictioncited by 9
- 2020Downstream Model Design of Pre-trained Language Model for Relation Extraction Taskcited by 7
- 2020Simplify-then-Translate: Automatic Preprocessing for Black-Box Machine Translationcited by 7
- 2020Learning from Easy to Complex: Adaptive Multi-curricula Learning for Neural Dialogue Generationcited by 6
- 2020Correction of Faulty Background Knowledge based on Condition Aware and Revise Transformer for Question Answeringcited by 1
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.