mlsanityData to 7 Oct 2026

The Inefficiency of Language Models in Scholarly Retrieval: An Experimental Walk-through

arXiv 2203.15364 ↗First posted 29 Mar 2022cs.IR cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Language models are increasingly becoming popular in AI-powered scientific IR systems. This paper evaluates popular scientific language models in handling (i) short-query texts and (ii) textual neighbors. Our experiments showcase the inability to retrieve relevant documents for a short-query text even under the most relaxed conditions. Additionally, we leverage textual neighbors, generated by small perturbations to the original text, to demonstrate that not all perturbations lead to close neighbors in the embedding space. Further, an exhaustive categorization yields several classes of orthographically and semantically related, partially related, and completely unrelated neighbors. Retrieval performance turns out to be more influenced by the surface form rather than the semantics of the text.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 34 references that are in the map. First posted on arXiv 29 Mar 2022.

Builds on

18 of 34 references.

References · 18

  1. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  2. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  3. 2014DeepWalk: Online Learning of Social Representationscited by 1,543
  4. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  5. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  6. 2014Distributed Representations of Sentences and Documentscited by 677
  7. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  8. 2020SPECTER: Document-level Representation Learning using Citation-informed Transformerscited by 426
  9. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  10. 2017struc2vec: Learning Node Representations from Structural Identitycited by 214
  11. 2021All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Qualitycited by 150
  12. 2020BioMegatron: Larger Biomedical Domain Language Modelcited by 76
6 more
  1. 2019Enhancing Clinical Concept Extraction with Contextual Embeddingscited by 48
  2. 2020ABNIRML: Analyzing the Behavior of Neural IR Modelscited by 23
  3. 2020Aspect-based Document Similarity for Research Paperscited by 21
  4. 2020Extracting a Knowledge Base of Mechanisms from COVID-19 Paperscited by 13
  5. 2017Paper2vec: Citation-Context Based Document Distributed Representation for Scholar Recommendationcited by 1
  6. 2021A dataset of mentorship in science with semantic and demographic estimationscited by 1

Built on by

Cited by year

Citing papers per year: 2022: 1.

1 paper.

  1. 2022A Computational Inflection for Scientific Discoverycited by 22