NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping

Yamin Li, Ange Lou, Ziyuan Xu, Shengchao Zhang, Shiyu Wang, Dario J. Englot, Soheil Kolouri, Daniel Moyer, Roza G. Bayrak, Catie Chang
Vanderbilt University
NeurIPS 2024
Paper arXiv Code (coming soon)

Our proposed NeuroBOLT can reconstruct fMRI timecourse from arbitrary region-of-interest from EEG, benefiting from the EEG representation learning from multi-dimensional aspects - spatial, temporal, and spectral.

Abstract

Functional magnetic resonance imaging (fMRI) is an indispensable tool in modern neuroscience, providing a non-invasive window into whole-brain dynamics at millimeter-scale spatial resolution. However, fMRI is constrained by issues such as high operation costs and immobility. With the rapid advancements in cross-modality synthesis and brain decoding, the use of deep neural networks has emerged as a promising solution for inferring whole-brain, high-resolution fMRI features directly from electroencephalography (EEG), a more widely accessible and portable neuroimaging modality. Nonetheless, the complex projection from neural activity to fMRI hemodynamic responses and the spatial ambiguity of EEG pose substantial challenges both in modeling and interpretability. Relatively few studies to date have developed approaches for EEG-fMRI translation, and although they have made significant strides, the inference of fMRI signals in a given study has been limited to a small set of brain areas and to a single condition (i.e., either resting-state or a specific task). The capability to predict fMRI signals in other brain areas, as well as to generalize across conditions, remain critical gaps in the field.

To tackle these challenges, we introduce a novel and generalizable framework: NeuroBOLT, i.e., Neuro-to-BOLD Transformer, which leverages multi-dimensional representation learning from temporal, spatial, and spectral domains to translate raw EEG data to the corresponding fMRI activity signals across the brain. Our experiments demonstrate that NeuroBOLT effectively reconstructs resting-state fMRI signals from primary sensory, high-level cognitive areas, and deep subcortical brain regions, achieving state-of-the-art accuracy and significantly advancing the integration of these two modalities.

Method

method

Results

Resting-state intra-subject prediction results. (A) The parcellation of the chosen ROIs. (B) The distribution of prediction performance (Pearson's correlation values), with example time-series reconstructions that represent performance levels near the mean (indicated by the small grey arrow in the histogram).

cross-subject results

Examples of reconstruction of the unseen scans. Scans with the median performance are shown.

cross-subject results


Please refer to the paper linked above for additional visualizations, as well as detailed quantitative and ablation studies.

BibTeX

@misc{li2024neuroboltrestingstateeegtofmrisynthesis,
      title={NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping}, 
      author={Yamin Li and Ange Lou and Ziyuan Xu and Shengchao Zhang and Shiyu Wang and Dario J. Englot and Soheil Kolouri and Daniel Moyer and Roza G. Bayrak and Catie Chang},
      year={2024},
      eprint={2410.05341},
      archivePrefix={arXiv},
      primaryClass={eess.IV},
      url={https://arxiv.org/abs/2410.05341}, 
}