Dense Syn-Net: Inter-Modal and Self-Guided Deep Learned PET-MR Reconstruction

We present a new framework for PET-MR reconstruction using deep learning. The proposed network named Dense Syn-Net is a new version of the synergistic network Syn-Net. This method is built on two model-based image reconstruction algorithms, the maximum a posteriori expectation-maximization algorithm for PET and the Landweber algorithm for MR, that we fully connect to each other. To avoid the use of handcrafted regularisations, the gradient of the priors for PET and MR are learned from training data along with regularisation strengths for both modalities. Two major modifications of the original Syn-Net have been introduced: i) iteration-dependent targets are used to ensure that the output of every module matches the corresponding iteration of the reconstruction of high quality data, ii) all the previous PET and MR estimates are used to guide the regularisation in a given module allowing both self and inter-modality guidance. Results on 2D simulated data show that Dense-Syn-Net outperforms conventional independent PET and MR reconstruction algorithms. For MR, our method offers improvements compared to deep learned independent methods and synergistic PET-MR reconstruction with mutually weighted quadratic priors. For PET reconstruction, our network shows greater robustness towards mismatches than MR-guided methods by better preserving modality-unique features. Dense Syn-Net improved global image reconstruction accuracy compared to Syn-Net, however the latter performs better for regions of mismatch. Future work will need to focus on assessing the performance of the network on 3D real data and performing ablation studies to investigate the need of fully connections.