Brain-OF

An OmniFunctional Foundation Model
for fMRI, EEG and MEG

Hanning Guo^1,2 Hanwen Bi^1,3 Farah Abdellatif^1,2 Andrei Galbenus^1,2
Jon. N. Shah^1,2,4 Abigail Morrison^1,2 Jürgen Dammers^1,2

⁴

HETEROGENEOUS SIGNALS · JOINT PRETRAINING

fMRIHemodynamic activity

EEGElectrical activity

MEGMagnetic activity

ARNESS · 128 SHARED LATENT TOKENSBrain-OFDINT attention + Sparse Mixture of Experts

ClassificationRegressionMultimodal fusion

Joint pretraining across modalities. Single- or paired-modality downstream inputs.

3functional modalities

37pretraining datasets

5.87Mpretraining samples

1.7Bparameters in Huge

WHY JOINT PRETRAINING?

Each modality measures a different aspect of brain activity. Together, they broaden both what the model can learn and the data it can learn from.

fMRI provides spatially precise measures of hemodynamic activity, whereas EEG and MEG resolve electrophysiological dynamics at high temporal resolution. Pretraining on a single modality limits access to this complementary information and to the collective scale of available brain recordings.

Brain-OF brings these heterogeneous datasets into a shared pretraining framework, combining spatial, temporal, and spectral diversity to learn transferable representations for unimodal and multimodal brain decoding.

02 / PRETRAINING CORPUS

Diverse brain signals.
Large-scale pretraining.

The pretraining corpus brings together resting-state and task recordings from 37 public datasets across international sites. It spans multicultural and multiracial populations, healthy participants and clinical cohorts, and diverse acquisition systems and experimental paradigms.

32,278unique participants

5,870,984pretraining samples

2.8 TBpretraining corpus size

Explore the pretraining datasets

Samples across datasets

Bar lengths use a logarithmic scale; exact sample counts are shown.

Acquisition diversity and exact dataset statistics

The corpus includes 14 EEG, 23 fMRI, and 5 MEG entries. Multimodal cohorts appear under each relevant modality. Channel counts range from 17 to 64 for EEG and 248 to 306 for MEG; fMRI uses 116 or 246 ROIs. Acquisition sampling rates and repetition times below are those reported in the original acquisition, before preprocessing.

Pretraining dataset statistics
Dataset	Modality	Paradigm	SR / TR	Channels / ROIs	Participants	Samples

03 / ARCHITECTURE

Different modalities.
A common representation.

Brain recordings differ in physical origin, sensor configurations or brain parcellations, and spatial, temporal, and frequency resolution. Brain-OF addresses resolution discrepancies, semantic heterogeneity, and neural-signal structure through three complementary components.

Bridge resolutions

ARNESS projects signals with heterogeneous spatial, temporal, and spectral resolutions into 128 shared latent tokens. This compact representation lets the backbone process diverse inputs and integrate paired modalities.

Resolve semantic heterogeneity

Sparse MoE combines shared experts for modality-invariant structure with routed experts for modality-specific semantics. DINT attention suppresses irrelevant context while integrating local and global dependencies.

Pretrain in both domains

Masked Temporal-Frequency Modeling (MTFM) jointly reconstructs signals and their Fourier-domain components, encouraging representations that capture complementary temporal and spectral structure.

Brain-OF pretraining, ARNESS sampling, serial multimodal fusion, Sparse MoE and DINT attention — **From heterogeneous signals to a shared backbone.** ARNESS also acts as the reconstruction decoder and supports serial multimodal fusion.

04 / RECONSTRUCTION QUALITY

Temporal dynamics.
Spectral and spatial structure.

MTFM couples signal reconstruction with Fourier-domain reconstruction during self-supervised pretraining. The examples below examine the resulting temporal dynamics, EEG spectral structure, fMRI functional connectivity, and MEG sensor topography.

MTFM: pretraining steps and objective

Transform
EEG & MEG: FFT along time, within each channel.
fMRI: FFT across the ordered ROIs, at each time point.
Mask Fourier components
Mask components in the transformed representation, then apply the inverse FFT to return the corrupted signal to its original layout.
Mask temporal segments
Mask segments of the corrupted signal and flatten the matrix into a channel/ROI–patch sequence for the model.
Reconstruct
Restore the decoder output to the original signal layout to recover the masked temporal segments and corresponding Fourier-domain components.

A joint reconstruction objective

The loss combines reconstruction errors in the temporal and Fourier domains. The weight $α$ balances their contributions.

ℒ_MTFM = (1 − α) ℒ_time + α ℒ_frequency

Both terms use Smooth L1 loss.

Temporal dynamics & spectral bursts

Compare the original and reconstructed EEG waveform and spectrogram. These qualitative examples show how temporal and spectral structure are represented together.

Brain-OF Huge · Qualitative reconstructions

05 / DOWNSTREAM EVALUATION

Brain-OF across
modalities and tasks.

Emotion recognition, seizure and abnormality detection, clinical classification, and age prediction. Explore all nine unimodal settings from the main comparison.

Choose an evaluation

Mean ± standard deviation over five seeds. Brain-OF uses full finetuning. “Best baseline” is selected from the 12 reported baselines for this setting.

Compare all methods for this evaluation


Method	Mean ± SD

FURTHER MODEL ANALYSIS

Adaptation, robustness, and representations

LoRA: efficient adaptation with low-rank updates≈2% trainable

LoRA freezes the pretrained backbone and learns low-rank update matrices. Brain-OF Base uses rank 32 and a scaling factor of 64, with approximately 2% of the total parameters trainable.

Full finetuning and LoRA on Brain-OF Base
Dataset	Modality	Metric	Full finetuning	LoRA
SEED-V	EEG	Cohen’s κ (%) ↑	23.98 ± 0.29	21.81 ± 0.13
TUAB	EEG	BAC (%) ↑	81.88 ± 0.17	81.30 ± 0.04
ADNI	fMRI	BAC (%) ↑	68.23 ± 2.07	68.44 ± 2.13
ADHD-200	fMRI	BAC (%) ↑	62.57 ± 1.19	62.10 ± 1.04
CamCAN	MEG	MAE (years) ↓	8.99 ± 0.14	9.17 ± 0.44
LEMON	fMRI	BAC (%) ↑	58.95 ± 2.13	59.36 ± 3.02

LoRA has slightly higher mean scores on ADNI and LEMON-fMRI. Full finetuning performs better on the other four settings, with the largest difference on SEED-V.

Linear probing and pretraining controlsRepresentation quality

Linear probing evaluates pretrained representation quality by freezing the entire backbone and training only a linear prediction head. The comparisons below also include full finetuning of pretrained models and full training of the same architecture from random initialization.

Brain-OF Base
Dataset	Modality	Metric	Full finetuning	Linear probing	From scratch
TUAB	EEG	BAC (%) ↑	81.88 ± 0.17	79.14 ± 0.24	77.59 ± 0.27
ADNI	fMRI	BAC (%) ↑	68.23 ± 2.07	65.22 ± 1.47	63.39 ± 2.36
CamCAN	MEG	MAE (years) ↓	8.99 ± 0.14	10.92 ± 0.22	9.69 ± 1.07

Brain-OF Huge
Dataset	Modality	Metric	Full finetuning	Linear probing	From scratch
TUAB	EEG	BAC (%) ↑	82.87 ± 0.18	81.92 ± 0.21	80.82 ± 0.18
ADNI	fMRI	BAC (%) ↑	71.79 ± 2.19	70.76 ± 1.22	64.85 ± 1.03
CamCAN	MEG	MAE (years) ↓	7.87 ± 0.22	9.02 ± 0.11	8.59 ± 0.94

Frozen pretrained features outperform training from scratch on TUAB and ADNI, while CamCAN age prediction benefits more from adapting the backbone.

Robustness under injected noise20 to −5 dB

DINT attention shows stronger noise robustness than vanilla attention in this stress test, retaining higher balanced accuracy as noise increases.

Balanced accuracy decreases for both methods as SNR falls from 20 to minus 5 dB. DINT has a higher mean at every tested noise level. Error bars show one standard deviation.

Representation space across scalesRaw inputs vs. Brain-OF

PCA highlights class-related separation in TUAB (EEG) and ADNI (fMRI), and continuous age-related structure in CamCAN (MEG), comparing raw inputs with finetuned Base, Large, and Huge embeddings.

Twelve PCA projections: raw inputs and finetuned Base, Large, and Huge embeddings for TUAB, ADNI, and CamCAN. Colors identify classes in TUAB and ADNI and chronological age in CamCAN.

Quantitative support for the representation space

First-component associations with the target strengthen from Base to Huge across these datasets. On TUAB, Base’s first component explains 73.0% of the embedding variance.

Correlations and class separability
Dataset	Modality	Model	Pearson r	Spearman ρ	AUC
TUAB	EEG	Base	0.701	0.678	0.892
TUAB	EEG	Large	0.707	0.687	0.898
TUAB	EEG	Huge	0.723	0.700	0.905
ADNI	fMRI	Base	0.380	0.388	0.724
ADNI	fMRI	Large	0.478	0.438	0.753
ADNI	fMRI	Huge	0.484	0.496	0.786
CamCAN	MEG	Base	0.746	0.735	—
CamCAN	MEG	Large	0.784	0.762	—
CamCAN	MEG	Huge	0.805	0.772	—

AUC uses the first principal component alone for classification. It is not applicable to the continuous age target in CamCAN.

06 / MULTIMODAL FUSION

When two views
work better together.

Serial ARNESS resampling combines paired modalities in the same latent representation, without an extra modality-specific fusion branch.

Mean ± SD, five seeds

07 / MODALITY CONTRIBUTIONS & INTERPRETABILITY

Where does the knowledge
come from?

How much does pretraining on each modality improve downstream prediction? Modality Shapley values quantify each modality’s average marginal contribution to downstream performance across combinations of pretraining modalities. Separately, occlusion analysis masks individual channels or ROIs to identify which signals most influence prediction performance.

Cross-modal contributions may exceed native ones.

On TUAB, the normalized modality Shapley values are 0.41 for MEG, 0.37 for fMRI, and 0.22 for EEG.

Task-level modality Shapley values and relative importance scores, with global summaries — Across evaluated tasks, fMRI has the largest global contribution under both scores.

A view of the signals behind the prediction

Occlusion maps for TUAB EEG, ADNI fMRI, and CamCAN MEG — **Model explanations across three modalities.** Occluding frontal/central EEG channels, temporal fMRI regions, and temporal/parietal MEG sensors affects prediction performance in the illustrated tasks.

08 / MODEL FAMILY

Scaling Brain-OF.
From Base to Giant.

Brain-OF spans compact and billion-parameter variants. Increasing model capacity expands the space for shared and specialized representations, while sparse routing keeps only a subset of parameters active for each token.

Cite this work

If you find Brain-OF useful in your research, please consider citing our work:

Brain-OF

Diverse brain signals.Large-scale pretraining.

Samples across datasets

Different modalities.A common representation.

Bridge resolutions

Resolve semantic heterogeneity

Pretrain in both domains

Temporal dynamics.Spectral and spatial structure.

Transform

Mask Fourier components

Mask temporal segments

Reconstruct