Catalyst Papers in Artificial Intelligence Research: A Landscape on ICLR from 2017 to 2025
On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment
Abstract (arXiv)
A small number of methodological contributions, including word2vec, the Transformer, large-scale pre-training, and reinforcement learning from human feedback, have reshaped NLP and AI research over the past decade. OpenReview now makes numeric reviewer scores and accept/reject decisions public for every ICLR submission. Whether such review signals identify trajectory-changing papers at submission time, however, remains untested at corpus scale. We answer this question on 36,113 papers from ICLR 2017--2025, identifying catalysts: papers whose descendants measurably redirect future research. We compare four disruptiveness measures (the Consolidation/Destabilization (CD) index, node2vec, the direction-aware Embedding Disruptiveness Measure (EDM), and an LLM-based semantic rater) and define a five-type operational catalyst taxonomy (topic initiator, topic bridge, within-topic redirector, simultaneous, and recognition-misaligned). EDM leads at identifying highly cited ICLR papers (AUC 0.83 vs. 0.60 for CD, 0.49 for node2vec, and 0.42 for the LLM rater). Topic initiators precede a 7.55× topic-share growth and topic bridges precede an 11.52× growth in cross-topic citation flow versus year-matched controls. We found that the peer review scores are essentially orthogonal to future disruptiveness (|ρ|≤0.005; accepted and rejected papers have indistinguishable mean EDM, p=0.11).
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 32 references that are in the map. First posted on arXiv 24 May 2026.
Builds on
13 of 32 references.
References · 13
- 2017Attention Is All You Needcited by 59,932
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
- 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
- 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
- 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
- 2022OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and conceptscited by 166
- 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 124
- 2021Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experimentcited by 49
- 2020An Open Review of OpenReview: A Critical Analysis of the Machine Learning Conference Review Processcited by 23
- 2015Defining and identifying Sleeping Beauties in sciencecited by 11
- 2023The disruption index is biased by citation inflationcited by 2
1 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Topical Phase Transitions in Artificial Intelligence Research: Large-Scale Evidence and an Early-Warning Signature for Emerging Topics
- 2026Do Reviewers Still Reward Lexical Complexity? A Frozen-Rater Study of Preference Drift in 124K ICLR Reviews
- 2026Paper Espresso: From Paper Overload to Research Insight
- 2025CS-PaperSum: A Large-Scale Dataset of AI-Generated Summaries for Scientific Papers
- 2026Benchmarking Agentic Review Systems
- 2026Does Multi-Agent Debate Improve AI Feedback on Research Papers?
- 2026The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishing
- 2025ELAIPBench: A Benchmark for Expert-Level Artificial Intelligence Paper Understanding