Syn-Net for Synergistic Deep-Learned PET-MR Reconstruction

We present in this work the first framework for synergistic PET-MR reconstruction using deep learning. The proposed network named Syn-Net is based on two interconnected unrolled regularised model-based image reconstruction algorithms. The maximum a posteriori expectation-maximisation algorithm is used for PET reconstruction and the regularised Landweber algorithm is used for MR reconstruction. Rather than using handcrafted regularisations, the gradient of the priors for MR and the entire regularisation step for PET are replaced by a neural network whose parameters are learnt from training data pairs. The regularisation strength for PET and MR are also learnt such that no hyperparameters are set by the user. The results on 2D simulated data showed that Syn-Net outperforms conventional independent PET and MR reconstruction algorithms by reducing noise and enhancing edges. The proposed network also removes aliasing artifacts in the reconstructed MR. Although Syn-Net is not the best deep learned method in terms of NRMSE on entire images, it outperforms conventional and independent deep learned MR-guided methods locally for regions of interest with mismatches between PET and MR. Future work will focus on comparing Syn-Net to conventional synergistic reconstruction methods, further exploring the benefits of the PET guidance for MR reconstruction in extreme cases of undersampling, investigating various interconnection strategies as well as network architectures and assessment for real data.