mlsanityData to 14 Oct 2026

Can LLMs Use Relational Transformer Embeddings?

arXiv 2609.00457 ↗First posted 31 Aug 2026cs.LGcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Injecting frozen relational-encoder embeddings as soft tokens into a large language model (LLM) is a conceptually appealing fusion strategy: the encoder handles multi-table structure, the LLM handles language and reasoning, and no lossy text serialization is required. We test this hypothesis concretely by injecting embeddings from a frozen Relational Transformer (RT) into Qwen3.5-4B via a learned MLP projection and LoRA adaptation, trained first with supervised fine-tuning (SFT) on chain-of-thought reasoning traces and then with group-based reinforcement learning (GSPO). We evaluate across 10 binary classification tasks on 6 relational databases from RelBench, under four supervision regimes: single-task (ST), within-dataset (WD), cross-dataset (CD), and all-task (ALL). The hybrid model does not consistently outperform standalone RT: it is frequently below random, highly sensitive to serialization format and relational-token budget, and unstable under RL training. We report these negative results and analyze the failure modes, arguing that soft-token fusion requires stronger alignment objectives and schema-aware design before it can serve as a reliable route to relational prediction.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 8 of its 10 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

8 of 10 references.

References · 8

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  4. 2024RelBench: A Benchmark for Deep Learning on Relational Databasescited by 96
  5. 2023Relational Deep Learning: Graph Representation Learning on Relational Databasescited by 56
  6. 2025Relational Transformer: Toward Zero-Shot Foundation Models for Relational Datacited by 39
  7. 2025BioReason: Incentivizing Multimodal Biological Reasoning within a DNA-LLM Modelcited by 37
  8. 2025Large Language Models are Good Relational Learnerscited by 24

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Embedder's Dilemma: LLMs Are Better, but at What Cost?
  2. 2026LLMs Explain't: A Post-Mortem on Semantic Interpretability in Transformer Models
  3. 2024LLMs for Relational Reasoning: How Far are We?
  4. 2025Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains
  5. 2026From Noise to Diversity: Random Embedding Injection in LLM Reasoning
  6. 2025TL;DR: Too Long, Do Re-weighting for Efficient LLM Reasoning Compression
  7. 2026Distilling Sequential Computation in Transformer Language Models
  8. 2026Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoders