# Join-Chain Network: A Logical Reasoning View of the Multi-head Attention in Transformer

arXiv: https://arxiv.org/abs/2210.02729 · HTML: https://mlsanity.com/paper/2210.02729/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 4 of its 10 references that are in the map.
- First posted on arXiv 6 Oct 2022.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 4 of 4)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Neural Logic Machines](https://mlsanity.com/paper/1904.11694/) (2019)
- [Lambda Dependency-Based Compositional Semantics](https://mlsanity.com/paper/1309.4408/) (2013)

## Built on by (most cited first, 2 of 2)

- [ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models](https://mlsanity.com/paper/2406.12044/) (2024)
- [MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning](https://mlsanity.com/paper/2409.06067/) (2024)

## Abstract (arXiv)

Developing neural architectures that are capable of logical reasoning has become increasingly important for a wide range of applications (e.g., natural language processing). Towards this grand objective, we propose a symbolic reasoning architecture that chains many join operators together to model output logical expressions. In particular, we demonstrate that such an ensemble of join-chains can express a broad subset of ''tree-structured'' first-order logical expressions, named FOET, which is particularly useful for modeling natural languages. To endow it with differentiable learning capability, we closely examine various neural operators for approximating the symbolic join-chains. Interestingly, we find that the widely used multi-head self-attention module in transformer can be understood as a special neural operator that implements the union bound of the join operator in probabilistic predicate space. Our analysis not only provides a new perspective on the mechanism of the pretrained models such as BERT for natural language understanding but also suggests several important future improvement directions.