An empirical comparison of different approaches for combining multimodal neuroimaging data with Support Vector Machine

In the pursuit of clinical utility, neuroimaging researchers of psychiatric and neurological illness are increasingly using analyses, such as support vector machine (SVM), that allow inference at the single-subject level. Recent studies employing single-modality data, however, suggest that classification accuracies must be improved for such utility to be realised. One possible solution is to integrate different data types to provide a single combined output classification; either by generating a single decision function based on an integrated kernel matrix, or, by creating an ensemble of multiple single modality classifiers and integrating their predictions. Here, we describe four integrative approaches: 1) an un-weighted sum of kernels, 2) multi-kernel learning, 3) prediction averaging, and 4) majority voting, and compare their ability to enhance classification accuracy relative to the best single-modality classification accuracy. We achieve this by integrating structural, functional and diffusion tensor magnetic resonance imaging data, in order to compare ultra-high risk (UHR; n=19), first episode psychosis (FEP; n=19) and healthy control subjects (HCs; n=19). Our results show that i) whilst integration can enhance classification accuracy by up to 13%, the frequency of such instances may be limited, ii) where classification can be enhanced, simple methods may yield greater increases relative to more computationally complex alternatives, and, iii) the potential for classification enhancement is highly influenced by the specific diagnostic comparison under consideration. In conclusion, our findings suggest that for moderately sized clinical neuroimaging datasets, combining different imaging modalities in a data-driven manner is no “magic bullet” for increasing classification accuracy.