# Learning representations of learning representations

arXiv: https://arxiv.org/abs/2404.08403 · HTML: https://mlsanity.com/paper/2404.08403/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 12 Apr 2024.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations](https://mlsanity.com/paper/2006.03659/) (2020)
- [Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings](https://mlsanity.com/paper/2202.06671/) (2022)

## Built on by (most cited first, 5 of 5)

- [Towards End-to-End Automation of AI Research](https://mlsanity.com/paper/2606.15497/) (2026)
- [Cropping outperforms dropout as an augmentation strategy for self-supervised training of text embeddings](https://mlsanity.com/paper/2508.03453/) (2025)
- [Who is a Better Matchmaker? Human vs. Algorithmic Judge Assignment in a High-Stakes Startup Competition](https://mlsanity.com/paper/2510.12692/) (2025)
- [NoveltyRank: A Retrieval-Augmented Framework for Conceptual Novelty Estimation in AI Research](https://mlsanity.com/paper/2512.14738/) (2025)
- [Dynamic Topic Modeling with a Higher-Order Hypergraphical Representation](https://mlsanity.com/paper/2605.28269/) (2026)

## Abstract (arXiv)

The ICLR conference is unique among the top machine learning conferences in that all submitted papers are openly available. Here we present the ICLR dataset consisting of abstracts of all 24 thousand ICLR submissions from 2017-2024 with meta-data, decision scores, and custom keyword-based labels. We find that on this dataset, bag-of-words representation outperforms most dedicated sentence transformer models in terms of kNN classification accuracy, and the top performing language models barely outperform TF-IDF. We see this as a challenge for the NLP community. Furthermore, we use the ICLR dataset to study how the field of machine learning has changed over the last seven years, finding some improvement in gender balance. Using a 2D embedding of the abstracts' texts, we describe a shift in research topics from 2017 to 2024 and identify hedgehogs and foxes among the authors with the highest number of ICLR submissions.