# Do Models Learn the Directionality of Relations? A New Evaluation: Relation Direction Recognition

arXiv: https://arxiv.org/abs/2105.09045 · HTML: https://mlsanity.com/paper/2105.09045/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 46 references that are in the map.
- First posted on arXiv 19 May 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019)
- [Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks](https://mlsanity.com/paper/1503.00075/) (2015)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [End-to-End Relation Extraction using LSTMs on Sequences and Tree Structures](https://mlsanity.com/paper/1601.00770/) (2016)
- [Graph Convolution over Pruned Dependency Trees Improves Relation Extraction](https://mlsanity.com/paper/1809.10185/) (2018)
- [Enriching Pre-trained Language Model with Entity Information for Relation Classification](https://mlsanity.com/paper/1905.08284/) (2019)
- [Attention Guided Graph Convolutional Networks for Relation Extraction](https://mlsanity.com/paper/1906.07510/) (2019)
- [SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals](https://mlsanity.com/paper/1911.10422/) (2019)
- [Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Path](https://mlsanity.com/paper/1508.03720/) (2015)
- [Classifying Relations by Ranking with Convolutional Neural Networks](https://mlsanity.com/paper/1504.06580/) (2015)
- [Relation Classification with Entity Type Restriction](https://mlsanity.com/paper/2105.08393/) (2021)
- [Improved Relation Classification by Deep Recurrent Neural Networks with Data Augmentation](https://mlsanity.com/paper/1601.03651/) (2016)
- [Interpreting Recurrent and Attention-Based Neural Models: a Case Study on Natural Language Inference](https://mlsanity.com/paper/1808.03894/) (2018)
- [Hierarchical Embeddings for Hypernymy Detection and Directionality](https://mlsanity.com/paper/1707.07273/) (2017)
- [Extracting Multiple-Relations in One-Pass with Pre-Trained Transformers](https://mlsanity.com/paper/1902.01030/) (2019)
- [TransGCN:Coupling Transformation Assumptions with Graph Convolutional Networks for Link Prediction](https://mlsanity.com/paper/1910.00702/) (2019)
- [Semantic Relation Classification via Convolutional Neural Networks with Simple Negative Sampling](https://mlsanity.com/paper/1506.07650/) (2015)
- [Relation Classification as Two-way Span-Prediction](https://mlsanity.com/paper/2010.04829/) (2020)
- [Downstream Model Design of Pre-trained Language Model for Relation Extraction Task](https://mlsanity.com/paper/2004.03786/) (2020)
- [Simplify-then-Translate: Automatic Preprocessing for Black-Box Machine Translation](https://mlsanity.com/paper/2005.11197/) (2020)
- [Learning from Easy to Complex: Adaptive Multi-curricula Learning for Neural Dialogue Generation](https://mlsanity.com/paper/2003.00639/) (2020)
- [Correction of Faulty Background Knowledge based on Condition Aware and Revise Transformer for Question Answering](https://mlsanity.com/paper/2006.16722/) (2020)

## Built on by (most cited first, 2 of 2)

- [When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challenges](https://mlsanity.com/paper/2401.10510/) (2024)
- [Reversing Arrows in Large Language Models](https://mlsanity.com/paper/2608.03512/) (2026)

## Abstract (arXiv)

Deep neural networks such as BERT have made great progress in relation classification. Although they can achieve good performance, it is still a question of concern whether these models recognize the directionality of relations, especially when they may lack interpretability. To explore the question, a novel evaluation task, called Relation Direction Recognition (RDR), is proposed to explore whether models learn the directionality of relations. Three metrics for RDR are introduced to measure the degree to which models recognize the directionality of relations. Several state-of-the-art models are evaluated on RDR. Experimental results on a real-world dataset indicate that there are clear gaps among them in recognizing the directionality of relations, even though these models obtain similar performance in the traditional metric (e.g. Macro-F1). Finally, some suggestions are discussed to enhance models to recognize the directionality of relations from the perspective of model design or training.