Multimodal Context-Aware Detection of Glioma Biomarkers Using MRI and WSI

The most malignant tumors of the central nervous system are adult-type diffuse gliomas. Historically, glioma subtype classification has been based on morphological features. However, since 2016, WHO recognizes that molecular evaluation is critical for subtyping. Among molecular markers, the mutation status of IDH1 and the codeletion of 1p/19q are crucial for the precise diagnosis of these malignancies. In pathology laboratories, however, manual screening for those markers is time-consuming and susceptible to error. To overcome these limitations, we propose a novel multimodal biomarker classification method that integrates image features derived from brain magnetic resonance imaging and histopathological exams. The proposed model consists of two branches, the first branch takes as input a multi-scale Hematoxylin and Eosin whole slide image, and the second branch uses the pre-segmented region of interest from the magnetic resonance imaging. Both branches are based on convolutional neural networks. After passing the exams by the two embedding branches, the output feature vectors are concatenated, and a multi-layer perceptron is used to classify the glioma biomarkers as a multi-class problem. In this work, several fusion strategies were studied, including a cascade model with mid-fusion; a mid-fusion model, a late fusion model, and a mid-context fusion model. The models were tested using a publicly available data set from The Cancer Genome Atlas. Our cross-validated classification models achieved an area under the curve of 0.874, 0.863, and 0.815 for the proposed multimodal, magnetic resonance imaging, and Hematoxylin and Eosin stain slide images respectively, indicating our multimodal model outperforms its unimodal counterparts and the state-of-the-art glioma biomarker classification methods.