'Orion-MSP' Brings Multi-Scale Sparse Attention to Tabular In-Context Learning

'Orion-MSP' Brings Multi-Scale Sparse Attention to Tabular In-Context Learning

Research · The Vault — AI Edition

The most valuable data inside most companies does not look like a paragraph of text or a photograph. It looks like a spreadsheet: rows of customers, columns of attributes, a database export of transactions, a CSV of sensor readings. By almost every estimate, the overwhelming majority of enterprise data is tabular — and yet the deep-learning revolution that reshaped language and vision largely passed it by. For structured data, gradient-boosted decision trees like XGBoost have remained the stubborn, hard-to-beat default, with neural networks rarely justifying the extra cost and tuning effort.

A new paper, Orion-MSP, is part of a fast-moving effort to change that. Released on arXiv (2511.02818) by Mohamed Bouadi, Pratinav Seth, Aditya Tanna, and Vinay Kumar Sankarapu of Lexsi Labs, it proposes a tabular foundation model built around multi-scale sparse attention — an architecture aimed squarely at one of the field's hardest unsolved problems: making in-context learning work efficiently on the wide, messy, high-dimensional tables that real businesses actually have.

Why tabular in-context learning matters

The breakthrough that made this line of work possible was TabPFN, a model from the University of Freiburg that showed transformers could perform in-context learning on tables. Instead of training a fresh model for each dataset, TabPFN is pre-trained once on millions of synthetic tables. At inference time you simply hand it your labeled rows and the rows you want predictions for; it produces answers in a single forward pass, often in under a second, with no per-dataset training and no hyperparameter tuning.

That promise — seconds instead of hours, strong results on small or messy data, and no tuning ritual — is why tabular foundation models have suddenly become one of the hottest corners of applied machine learning. The commercial stakes are real: SAP recently made a roughly billion-euro bet on Prior Labs, the startup commercializing TabPFN.

But the original approach has well-known ceilings. TabPFN-style models historically struggled to scale beyond a few thousand samples and a modest number of features. The dense attention mechanism at their core scales quadratically with table width, so very wide tables — hundreds or thousands of columns — become expensive fast. That is precisely where Orion-MSP focuses its effort.

What Orion-MSP changes

The authors identify three limitations in current tabular architectures: single-scale feature processing that ignores hierarchical structure; dense attention that scales quadratically with the number of features; and strictly sequential component processing that blocks iterative refinement and cross-component communication. Orion-MSP attacks all three.

Its central idea is multi-scale sparse attention. Rather than letting every feature attend to every other feature at one resolution, the model processes features at three scales — described in the paper as scales 1, 4, and 16 — using a mix of windowed (local), global, and random attention patterns. Borrowing ideas proven in efficient long-sequence transformers, this design reduces attention's cost from quadratic toward near-linear in the number of features, while a hierarchical view lets the model capture everything from individual cell-level signals up to dataset-wide patterns and feature groups.

The second ingredient is a Perceiver-style memory — a compact, cross-component store that compresses dataset-level information so it can be shared efficiently across samples and components. Together, the authors argue, these let Orion-MSP refine its representations iteratively and communicate across the model's parts rather than passing information in a single sequential chain.

How well does it work?

The authors evaluate Orion-MSP across more than 100 datasets drawn from three established benchmark suites — TALENT, TabZilla, and OpenML-CC18 — comparing against leading tabular foundation models including the TabPFN family and TabICL.

The headline results are competitive rather than dominant, and the paper is candid about it. On the TALENT suite, the reported accuracies sit close together: roughly 0.85 for TabPFN, 0.847 for TabICL, and 0.846 for Orion-MSP. Across the full evaluation, the TabPFN family posts the best aggregate numbers, with mean accuracies in the 0.85–0.88 range, while Orion-MSP follows closely — the authors report overall ranks of about 4.61 in the zero-shot setting and 2.26 under meta-learning. (Where specific figures cannot be independently verified, they should be read as results reported by the authors.)

The more important claim is not that Orion-MSP wins outright, but where it wins. The authors state it delivers state-of-the-art zero-shot performance particularly on imbalanced, high-dimensional, and context-rich datasets, with stronger calibration stability than both TabPFN and TabICL. In other words, the architecture is designed to hold up exactly where the older models strain: wide tables, skewed class distributions, and data-rich enterprise settings.

Why this matters for enterprise AI

If tabular foundation models live up to their billing, they change the economics of a vast amount of unglamorous but high-value work — churn prediction, fraud scoring, demand forecasting, credit risk, quality control. The pitch is a single model that ingests a company's table and returns calibrated predictions without a data-science team spending days on feature engineering and tuning.

Scalability is the gating factor for that vision, and width is a particular pain point because real enterprise tables are often very wide. By targeting near-linear scaling in feature count, Orion-MSP is explicitly engineering for the shape of real business data rather than the tidy academic benchmarks where these models first proved themselves. Lexsi Labs is also building out tooling around the model family, including an open library, TabTune, for inference and fine-tuning across tabular foundation models — a sign the work is aimed at practitioners, not just leaderboards.

What to watch

Three things will determine whether Orion-MSP and its siblings matter beyond the arXiv listing. First, can sparse-attention designs translate near-linear theoretical scaling into real wall-clock and memory wins on genuinely large, wide tables? Second, will tabular foundation models reliably beat well-tuned gradient-boosted trees in production, not just in aggregate benchmark averages? And third, how quickly does this crowded research area — TabPFN, TabICL, and now multiple Orion variants including a bi-axial-attention sibling, Orion-Bix — consolidate into models that practitioners actually deploy. Orion-MSP is a credible entry in that race, and its bet on multi-scale sparsity is one of the clearer answers yet to the field's scaling problem.

---

Sources: arXiv 2511.02818; Lexsi Labs; Hugging Face (Lexsi/Orion-MSP); TabPFN (University of Freiburg) and TabICL research.