mlsanityData to 7 Oct 2026

Do Language Embeddings Capture Scales?

arXiv 2010.05345 ↗First posted 11 Oct 2020cs.CLcited by 63

Not on the map. Why?

Abstract (arXiv)

Pretrained Language Models (LMs) have been shown to possess significant linguistic, common sense, and factual knowledge. One form of knowledge that has not been studied yet in this context is information about the scalar magnitudes of objects. We show that pretrained language models capture a significant amount of this information but are short of the capability required for general common-sense reasoning. We identify contextual information in pre-training and numeracy as two key factors affecting their performance and show that a simple method of canonicalizing numbers can have a significant effect on the results.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 21; 2023: 14; 2024: 10; 2025: 3; 2026: 3.

63 papers.

Sort:

By area:

  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  2. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  3. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  4. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  5. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  6. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  7. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  8. 2022MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Datacited by 120
  9. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  10. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  11. 2023GeoLLM: Extracting Geospatial Knowledge from Large Language Modelscited by 62
  12. 2021TIMEDIAL: Temporal Commonsense Reasoning in Dialogcited by 55
38 more
  1. 2021What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structurecited by 52
  2. 2022FiNER: Financial Numeric Entity Recognition for XBRL Taggingcited by 47
  3. 2022Things not Written in Text: Exploring Spatial Commonsense from Visual Signalscited by 41
  4. 2021MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solvingcited by 32
  5. 2022GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Modelscited by 30
  6. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  7. 2023Mathematical Language Models: A Surveycited by 24
  8. 2021How Much Coffee Was Consumed During EMNLP 2019? Fermi Problems: A New Reasoning Challenge for AIcited by 22
  9. 2021Probing artificial neural networks: insights from neurosciencecited by 17
  10. 2022NumHTML: Numeric-Oriented Hierarchical Transformer Model for Multi-task Financial Forecastingcited by 16
  11. 2021NumGPT: Improving Numeracy Ability of Generative Pre-trained Modelscited by 15
  12. 2022"John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibilitycited by 15
  13. 2023Evaluating Transformer Language Models on Arithmetic Operations Using Number Decompositioncited by 15
  14. 2024What Do Language Models Hear? Probing for Auditory Representations in Language Modelscited by 15
  15. 2021Investigating Numeracy Learning Ability of a Text-to-Text Transfer Modelcited by 13
  16. 2022Lila: A Unified Benchmark for Mathematical Reasoningcited by 13
  17. 2023Understanding Finetuning for Factual Knowledge Extraction from Language Modelscited by 12
  18. 2023TaskLAMA: Probing the Complex Task Understanding of Language Modelscited by 12
  19. 2024Regress, Don't Guess -- A Regression-like Loss on Number Tokens for Language Modelscited by 12
  20. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  21. 2022Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Modelscited by 10
  22. 2023Human Behavioral Benchmarking: Numeric Magnitude Comparison Effects in Large Language Modelscited by 10
  23. 2022Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarkscited by 8
  24. 2022LUNA: Language Understanding with Number Augmentations on Transformers via Number Plugins and Pre-trainingcited by 8
  25. 2023BERT4CTR: An Efficient Framework to Combine Pre-trained Language Model with Non-textual Features for CTR Predictioncited by 8
  26. 2024GRASP: A Grid-Based Benchmark for Evaluating Commonsense Spatial Reasoningcited by 7
  27. 2026EHR-RAG: Bridging Long-Horizon Structured Electronic Health Records and Large Language Models via Enhanced Retrieval-Augmented Generationcited by 6
  28. 2021Masked Measurement Prediction: Learning to Jointly Predict Quantities and Units from Textual Contextcited by 5
  29. 2022Transformers and the representation of biomedical background knowledgecited by 5
  30. 2023Learning Non-linguistic Skills without Sacrificing Linguistic Proficiencycited by 5
  31. 2024How to Leverage Digit Embeddings to Represent Numbers?cited by 5
  32. 2024Interleaving Text and Number Embeddings to Solve Mathemathics Problemscited by 5
  33. 2021What's the best place for an AI conference, Vancouver or ______: Why completing comparative questions is difficultcited by 4
  34. 2021Injecting Numerical Reasoning Skills into Knowledge Base Question Answering Modelscited by 4
  35. 2022Overcoming Barriers to Skill Injection in Language Modeling: Case Study in Arithmeticcited by 4
  36. 2023Probing Physical Reasoning with Counter-Commonsense Contextcited by 4
  37. 2023Can NLP Models 'Identify', 'Distinguish', and 'Justify' Questions that Don't have a Definitive Answer?cited by 4
  38. 2025Number Representations in LLMs: A Computational Parallel to Human Perceptioncited by 4

50 most cited of 63 All 63 (JSON)