Semantic Virtual Shadows (SVS) for Improved Perception in 4D OCT Guided Surgery

Swept-Source Optical Coherence Tomography (SS-OCT) integrated with surgical microscopes has enabled fast, high-resolution, and volumetric visualization of delicate tissue-instrument interactions. However, some visual features, which provide essential perceptual information in microscopic surgery, are not present in 4D OCT. Such a feature is the shadow of the surgical instruments cast onto the retina by the endo-illumination probe, which is among the most important cognitive cues for perceptual distance estimation. In this work, we propose Semantic Virtual Shadows (SVS), a novel concept to artificially generate instrument-specific shadows in OCT volumes, enabling naturally non-existent but important perceptual cues that are present in microscopic surgery. Semantic scene information is leveraged by considering only voxels associated with shadow-casting and shadow-receiving objects, identified using a learning-based approach and efficient volume processing, respectively. Real-time performance is achieved by a precomputed semantic shadow volume texture that assigns a shadowing factor to each voxel associated with a shadow-receiving object. The novelty of the method includes not only instrument-specific shadowing on the surface anatomy but also exclusively on deep-seated subsurface structures, providing advantages for various vitreoretinal procedures. Our user study indicates the benefits of the method for 4D OCT-guided surgery in several cognitive and performance-specific aspects.