# Implementing Derivations of Definite Logic Programs with Self-Attention Networks

arXiv: https://arxiv.org/abs/2410.11396 · HTML: https://mlsanity.com/paper/2410.11396/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 1 of its 7 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 1 of 1)

- [Theoretical Limitations of Self-Attention in Neural Sequence Models](https://mlsanity.com/paper/1906.06755/) (2019)

## Abstract (arXiv)

In this paper we propose that a restricted version of logical inference can be implemented with self-attention networks. We are aiming at showing that LLMs (Large Language Models) constructed with transformer networks can make logical inferences. We would reveal the potential of LLMs by analyzing self-attention networks, which are main components of transformer networks. Our approach is not based on semantics of natural languages but operations of logical inference. %point of view. We show that hierarchical constructions of self-attention networks with feed forward networks (FFNs) can implement top-down derivations for a class of logical formulae. We also show bottom-up derivations are also implemented for the same class. We believe that our results show that LLMs implicitly have the power of logical inference.