# On the Use of Unrealistic Predictions in Hundreds of Papers Evaluating Graph Representations

arXiv: https://arxiv.org/abs/2112.04274 · HTML: https://mlsanity.com/paper/2112.04274/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 32 references that are in the map.
- First posted on arXiv 8 Dec 2021.

## Builds on (most cited first, 13 of 13)

- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [LINE: Large-scale Information Network Embedding](https://mlsanity.com/paper/1503.03578/) (2015)
- [Network Embedding as Matrix Factorization: Unifying DeepWalk, LINE, PTE, and node2vec](https://mlsanity.com/paper/1710.02971/) (2017)
- [AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification](https://mlsanity.com/paper/1811.01727/) (2018)
- [Bonsai -- Diverse and Shallow Trees for Extreme Multi-label Classification](https://mlsanity.com/paper/1904.08249/) (2019)
- [A Unified View of Multi-Label Performance Measures](https://mlsanity.com/paper/1609.00288/) (2016)
- [Extreme Multi-label Learning for Semantic Matching in Product Search](https://mlsanity.com/paper/2106.12657/) (2021)
- [InfiniteWalk: Deep Network Embeddings as Laplacian Embeddings with a Nonlinearity](https://mlsanity.com/paper/2006.00094/) (2020)
- [A Comparative Study for Unsupervised Network Representation Learning](https://mlsanity.com/paper/1903.07902/) (2019)
- [LASAGNE: Locality And Structure Aware Graph Node Embedding](https://mlsanity.com/paper/1710.06520/) (2017)
- [Investigating Extensions to Random Walk Based Graph Embedding](https://mlsanity.com/paper/2002.07252/) (2020)
- [Parameter Selection: Why We Should Pay More Attention to It](https://mlsanity.com/paper/2107.05393/) (2021)

## Built on by (most cited first, 1 of 1)

- [Linear Classifier: An Often-Forgotten Baseline for Text Classification](https://mlsanity.com/paper/2306.07111/) (2023)

## Abstract (arXiv)

Prediction using the ground truth sounds like an oxymoron in machine learning. However, such an unrealistic setting was used in hundreds, if not thousands of papers in the area of finding graph representations. To evaluate the multi-label problem of node classification by using the obtained representations, many works assume in the prediction stage that the number of labels of each test instance is known. In practice such ground truth information is rarely available, but we point out that such an inappropriate setting is now ubiquitous in this research area. We detailedly investigate why the situation occurs. Our analysis indicates that with unrealistic information, the performance is likely over-estimated. To see why suitable predictions were not used, we identify difficulties in applying some multi-label techniques. For the use in future studies, we propose simple and effective settings without using practically unknown information. Finally, we take this chance to conduct a fair and serious comparison of major graph-representation learning methods on multi-label node classification.