mlsanityData to 14 Oct 2026

LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning

arXiv 2101.06223 ↗First posted 15 Jan 2021cs.LG cs.AI cs.LOcited by 46

Not on the map. Why?

Abstract (arXiv)

While designing inductive bias in neural architectures has been widely studied, we hypothesize that transformer networks are flexible enough to learn inductive bias from suitable generic tasks. Here, we replace architecture engineering by encoding inductive bias in the form of datasets. Inspired by Peirce's view that deduction, induction, and abduction are the primitives of reasoning, we design three synthetic tasks that are intended to require the model to have these three abilities. We specifically design these tasks to be synthetic and devoid of mathematical knowledge to ensure that only the fundamental reasoning biases can be learned from these tasks. This defines a new pre-training methodology called "LIME" (Learning Inductive bias for Mathematical rEasoning). Models trained with LIME significantly outperform vanilla transformers on four very different large mathematical reasoning benchmarks. Unlike dominating the computation cost as traditional pre-training approaches, LIME requires only a small fraction of the computation cost of the typical downstream task. The code for generating LIME tasks is available at https://github.com/tonywu95/LIME.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 42 of its 76 references that are in the map. First posted on arXiv 15 Jan 2021.

Builds on

42 of 76 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  10. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  12. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
30 more
  1. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  2. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  3. 2020Generative Language Modeling for Automated Theorem Provingcited by 388
  4. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 347
  5. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  6. 2019Deep Learning for Symbolic Mathematicscited by 270
  7. 2018Learning a SAT Solver from Single-Bit Supervisioncited by 200
  8. 2019What Can Neural Networks Reason About?cited by 159
  9. 2019The Lean mathematical librarycited by 134
  10. 2019Learning to Prove Theorems via Interacting with Proof Assistantscited by 109
  11. 2019Guiding High-Performance SAT Solvers with Unsat-Core Predictionscited by 62
  12. 2019Graph Representations for Higher-Order Logic and Theorem Provingcited by 55
  13. 2018GamePad: A Learning Environment for Theorem Provingcited by 48
  14. 2017Premise Selection for Theorem Proving by Deep Graph Embeddingcited by 44
  15. 2018TacticToe: Learning to Prove with Tacticscited by 44
  16. 2020INT: An Inequality Benchmark for Evaluating Generalization in Theorem Provingcited by 44
  17. 2019Exploration of Neural Machine Translation in Autoformalization of Mathematics in Mizarcited by 40
  18. 2019Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solvingcited by 35
  19. 2020Can neural networks acquire a structural bias from raw linguistic data?cited by 35
  20. 2020Universal linguistic inductive biases via meta-learningcited by 29
  21. 2019Learning to Reason in Large Theories without Imitationcited by 27
  22. 2020First Neural Conjecturing Datasets and Experimentscited by 26
  23. 2018Can Neural Networks Understand Logical Entailment?cited by 24
  24. 2019Improving Graph Neural Network Representations of Logical Formulae with Subgraph Poolingcited by 20
  25. 2018Learning Heuristics for Quantified Boolean Formulas through Deep Reinforcement Learningcited by 15
  26. 2020Enhancing SAT solvers with glue variable predictionscited by 11
  27. 2020Learning Branching Heuristics for Propositional Model Countingcited by 9
  28. 2019Formalising perfectoid spacescited by 7
  29. 2020IsarStep: a Benchmark for High-level Mathematical Reasoningcited by 7
  30. 2019Guiding Inferences in Connection Tableau by Recurrent Neural Networkscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 9; 2023: 13; 2024: 5; 2025: 8; 2026: 4.

46 papers.

Sort:

By area:

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  3. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 427
  4. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  5. 2022Language models show human-like content effects on reasoning taskscited by 173
  6. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  7. 2021Proof Artifact Co-training for Theorem Proving with Language Modelscited by 132
  8. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  9. 2023Large Language Models as Analogical Reasonerscited by 107
  10. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  11. 2022Vision Transformers provably learn spatial structurecited by 93
  12. 2021NaturalProofs: Mathematical Theorem Proving in Natural Languagecited by 81
34 more
  1. 2022Language Models Can Teach Themselves to Program Bettercited by 78
  2. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 75
  3. 2024Lean-STaR: Learning to Interleave Thinking and Provingcited by 68
  4. 2024A Survey on Deep Learning for Theorem Provingcited by 57
  5. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 40
  6. 2023Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AIcited by 32
  7. 2022Insights into Pre-training via Simpler Synthetic Taskscited by 26
  8. 2021Symbolic Behaviour in Artificial Intelligencecited by 25
  9. 2021Unsupervised Neural Machine Translation with Generative Language Models Onlycited by 24
  10. 2023TRIGO: Benchmarking Formal Mathematical Proof Reduction for Generative Language Modelscited by 24
  11. 2023Mathematical Language Models: A Surveycited by 24
  12. 2023ExPT: Synthetic Pretraining for Few-Shot Experimental Designcited by 22
  13. 2021Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLPcited by 16
  14. 2023Through the Lens of Core Competency: Survey on Evaluation of Large Language Modelscited by 12
  15. 2025General Intelligence Requires Reward-based Pretrainingcited by 12
  16. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  17. 2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
  18. 2022A Survey in Mathematical Language Processingcited by 9
  19. 2023Pre-training with Synthetic Data Helps Offline Reinforcement Learningcited by 9
  20. 2025A Survey of Inductive Reasoning for Large Language Modelscited by 8
  21. 2025Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformerscited by 8
  22. 2023SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulationcited by 7
  23. 2024Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Modelscited by 6
  24. 2025AutoMathKG: The automated mathematical knowledge graph based on LLM and vector databasecited by 6
  25. 2025A Theoretical Framework for OOD Robustness in Transformers using Gevrey Classescited by 4
  26. 2023Thrust: Adaptively Propels Large Language Models with External Knowledgecited by 3
  27. 2024Scaling Synthetic Logical Reasoning Datasets with Context-Sensitive Declarative Grammarscited by 3
  28. 2025Transfer of Structural Knowledge from Synthetic Languagescited by 3
  29. 2026Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Resultscited by 3
  30. 2023Enhancing Neural Theorem Proving through Data Augmentation and Dynamic Sampling Methodcited by 2
  31. 2022Big Learningcited by 0
  32. 2024EvoGPT-f: An Evolutionary GPT Framework for Benchmarking Formal Math Languagescited by 0
  33. 2026Inductive Generalization for Robotic Manipulationcited by 0
  34. 2026Lasting Effects of Abstract Pretraining Beyond Perplexitycited by 0