Uncertainty Estimation and Visualization for Multi-modal Image Segmentation

Abstract

Multi-modal imaging allows for the integration of complementary information from multiple medical imaging modalities for an improved analysis. The multiple information channels may lead to a reduction of the uncertainty in the analysis and decision-making process. Recently, efforts have been made to estimate the uncertainty in unimodal image segmentation decisions and visually convey this information to the medical experts that examine the image segmentation results. We propose an approach to extend uncertainty estimation and visualization methods to multi-modal image segmentations. We combine probabilistic uni-modal image segmentation results using the concept of ensemble of classifiers. The uncertainty is computed using a measure that is based on the Kullback- Leibler divergence. We apply our approach for an improved segmentation of Multiple Sclerosis (MS) lesions from multiple MR brain imaging modalities. Moreover, we demonstrate how our approach can be used to estimate and visualize the growth of a brain tumor area for imaging data taken at multiple points in time. Both the MS lesion and the area of tumor growth are detected as areas of high uncertainty due to different characteristics in different imaging modalities and changes over time, respectively.