# Orion-Bix: Bi-Axial Attention for Tabular In-Context Learning

arXiv: https://arxiv.org/abs/2512.00181 · HTML: https://mlsanity.com/paper/2512.00181/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 28 Nov 2025.

## Builds on (most cited first, 17 of 17)

- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabICL: A Tabular Foundation Model for In-Context Learning on Large Data](https://mlsanity.com/paper/2502.05564/) (2025)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [OpenML Benchmarking Suites](https://mlsanity.com/paper/1708.03731/) (2017)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [Why Tabular Foundation Models Should Be a Research Priority](https://mlsanity.com/paper/2405.01147/) (2024)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [Large Scale Transfer Learning for Tabular Data via Language Modeling](https://mlsanity.com/paper/2406.12031/) (2024)
- [Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks](https://mlsanity.com/paper/2310.19909/) (2023)
- [Natural Language Reasoning, A Survey](https://mlsanity.com/paper/2303.14725/) (2023)
- [A Closer Look at Deep Learning Methods on Tabular Datasets](https://mlsanity.com/paper/2407.00956/) (2024)
- [When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models](https://mlsanity.com/paper/2402.11100/) (2024)
- [Orion-MSP: Multi-Scale Sparse Attention for Tabular In-Context Learning](https://mlsanity.com/paper/2511.02818/) (2025)
- [TabTune: A Unified Library for Inference and Fine-Tuning Tabular Foundation Models](https://mlsanity.com/paper/2511.02802/) (2025)
- [AssistRAG: Boosting the Potential of Large Language Models with an Intelligent Information Assistant](https://mlsanity.com/paper/2411.06805/) (2024)

## Built on by (most cited first, 10 of 10)

- [Exploring Fine-Tuning for Tabular Foundation Models](https://mlsanity.com/paper/2601.09654/) (2026)
- [Shaping the Prior: How Synthetic Task Distributions Determine Tabular Foundation Model Quality](https://mlsanity.com/paper/2605.18971/) (2026)
- [Pocket Foundation Models: Distilling TFMs into CPU-Ready Gradient-Boosted Trees](https://mlsanity.com/paper/2605.18654/) (2026)
- [Distilling Tabular Foundation Models for Structured Health Data](https://mlsanity.com/paper/2605.18702/) (2026)
- [Data Presentation Over Architecture: Resampling Strategies for Credit Risk Prediction with Tabular Foundation Models](https://mlsanity.com/paper/2605.18635/) (2026)
- [Ensembling Tabular Foundation Models - A Diversity Ceiling And A Calibration Trap](https://mlsanity.com/paper/2605.18696/) (2026)
- [From Behavior to Provenance: Attributing Tabular Foundation Models to Synthetic Pretraining Data](https://mlsanity.com/paper/2610.02347/) (2026)
- [When Does Synthetic Relational Data Teach Models to Use Relations? Tracing Predictive Structure from Pretraining Data to Model Behavior](https://mlsanity.com/paper/2610.03057/) (2026)
- [Tabular Foundation Models Are Effectively Shallow](https://mlsanity.com/paper/2610.13253/) (2026)
- [OrionMoE-v1.0: A Mixture-of-Experts Foundation Model for Tabular In-Context Learning](https://mlsanity.com/paper/2610.20418/) (2026)

## Abstract (arXiv)

Tabular data drive most real-world machine learning applications, yet building general-purpose models for them remains difficult. Mixed numeric and categorical fields, weak feature structure, and limited labeled data make scaling and generalization challenging. To this end, we introduce Orion-Bix, a tabular foundation model that combines biaxial attention with meta-learned in-context reasoning for few-shot tabular learning. Its encoder alternates standard, grouped, hierarchical, and relational attention, fusing their outputs through multi-CLS summarization to capture both local and global dependencies efficiently. A label-aware ICL head adapts on the fly and scales to large label spaces via hierarchical decision routing. Meta-trained on synthetically generated, structurally diverse tables with causal priors, Orion-Bix learns transferable inductive biases across heterogeneous data. Delivered as a scikit-learn compatible foundation model, it outperforms gradient-boosting baselines and remains competitive with state-of-the-art tabular foundation models on public benchmarks, showing that biaxial attention with episodic meta-training enables robust, few-shot-ready tabular learning. The model is publicly available at https://github.com/Lexsi-Labs/Orion-BiX .