mlsanityData to 14 Oct 2026

Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving

arXiv 1910.06611 ↗First posted 15 Oct 2019cs.LG stat.MLcited by 35

Not on the map. Why?

Abstract (arXiv)

We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure. Our Tensor-Product Transformer (TP-Transformer) sets a new state of the art on the recently-introduced Mathematics Dataset containing 56 categories of free-form math word-problems. The essential component of the model is a novel attention mechanism, called TP-Attention, which explicitly encodes the relations between each Transformer cell and the other cells from which values have been retrieved by attention. TP-Attention goes beyond linear combination of retrieved values, strengthening representation-building and resolving ambiguities introduced by multiple layers of standard attention. The TP-Transformer's attention maps give better insights into how it is capable of solving the Mathematics Dataset's challenging problems. Pretrained models and code will be made available after publication.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 10; 2022: 8; 2023: 3; 2024: 8; 2025: 3; 2026: 3.

35 papers.

Sort:

By area:

  1. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  2. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
  3. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  4. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  5. 2022Causal Transformer for Estimating Counterfactual Outcomescited by 123
  6. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  7. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 89
  8. 2021LIME: Learning Inductive Bias for Primitives of Mathematical Reasoningcited by 46
  9. 2022Neurocompositional computing: From the Central Paradox of Cognition to a new generation of AI systemscited by 44
  10. 2022Formal Specifications from Natural Languagecited by 33
  11. 2021Compositional Processing Emerges in Neural Networks Solving Math Problemscited by 11
  12. 2021Enriching Transformers with Structured Tensor-Product Representations for Abstractive Summarizationcited by 11
23 more
  1. 2022Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Modelscited by 10
  2. 2021Learning Symbolic Rules for Reasoning in Quasi-Natural Languagecited by 9
  3. 2024Probabilistic Abduction for Visual Abstract Reasoning via Learning Rules in Vector-symbolic Architecturescited by 9
  4. 2024Towards Learning Abductive Reasoning using VSA Distributed Representationscited by 9
  5. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 8
  6. 2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
  7. 2023LLMs as Potential Brainstorming Partners for Math and Science Problemscited by 6
  8. 2021Generating Symbolic Reasoning Problems with Transformer GANscited by 5
  9. 2022Structural Biases for Improving Transformers on Translation into Morphologically Rich Languagescited by 5
  10. 2023Differentiable Tree Operations Promote Compositional Generalizationcited by 5
  11. 2022Enhancing Neural Mathematical Reasoning by Abductive Combination with Symbolic Librarycited by 4
  12. 2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
  13. 2024Arctic-TILT. Business Document Understanding at Sub-Billion Scalecited by 3
  14. 2025Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environmentscited by 3
  15. 2021A Reinforcement Learning Environment for Mathematical Reasoning via Program Synthesiscited by 2
  16. 2022Semantic Representations of Mathematical Expressions in a Continuous Vector Spacecited by 2
  17. 2024Attention-based Iterative Decomposition for Tensor Product Representationcited by 2
  18. 2024Terminating Differentiable Tree Expertscited by 1
  19. 2026Tensor Product Representation Probes Reveal Shared Structure Across Linear Directionscited by 1
  20. 2026The Emergent Symbolic Structure of Artificial Neural Networkscited by 1
  21. 2022A Short Survey of Systematic Generalizationcited by 0
  22. 2024Discrete Dictionary-based Decomposition Layer for Structured Representation Learningcited by 0
  23. 2026TANGO: Treating Tokens as Operatorscited by 0