# Mixture of Latent Experts Using Tensor Products

arXiv: https://arxiv.org/abs/2405.16671 · HTML: https://mlsanity.com/paper/2405.16671/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2024.

## Built on by (most cited first, 4 of 4)

- [DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution](https://mlsanity.com/paper/2405.17357/) (2024)
- [MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation](https://mlsanity.com/paper/2406.17484/) (2024)
- [Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models](https://mlsanity.com/paper/2408.15915/) (2024)
- [Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs](https://mlsanity.com/paper/2606.31413/) (2026)

## Abstract (arXiv)

In multi-task learning, the conventional approach involves training a model on multiple tasks simultaneously. However, the training signals from different tasks can interfere with one another, potentially leading to textitnegative transfer. To mitigate this, we investigate if modular language models can facilitate positive transfer and systematic generalization. Specifically, we propose a novel modular language model (textttTensorPoly), that balances parameter efficiency with nuanced routing methods. For textitmodules, we reparameterize Low-Rank Adaptation (textttLoRA) by employing an entangled tensor through the use of tensor product operations and name the resulting approach textttTLoRA. For textitrouting function, we tailor two innovative routing functions according to the granularity: textttTensorPoly-I which directs to each rank within the entangled tensor while textttTensorPoly-II offers a finer-grained routing approach targeting each order of the entangled tensor. The experimental results from the multi-task T0-benchmark demonstrate that: 1) all modular LMs surpass the corresponding dense approaches, highlighting the potential of modular language models to mitigate negative inference in multi-task learning and deliver superior outcomes. 2) textttTensorPoly-I achieves higher parameter efficiency in adaptation and outperforms other modular LMs, which shows the potential of our approach in multi-task transfer learning.