Ship Detection and Visual Augmentation during Helicopter Ship Deck Operations

Shipboard approach and recovery missions of helicopters involves operational and sensory challenges within the helicopter ship interface. Degraded Visual Environments (DVEs) in the harsh maritime environment and unsteady ship motion make it difficult for the landing helicopter to detect and approach the moving platform. Therefore, technological aids that enhance the pilot’s visual cueing environment are considered within this paper. An image-based ship detection method and a Helmet Mounted Display (HMD) visual augmentation concept allows insights into a state-of-the-art detection and augmentation design for an “image of the outside world” in the manner that pilots prefer for naval operations. The goal of this research is to investigate in advanced detection and visual augmentation techniques in the form of a pilot assistance system for rotorcraft systems during helicopter shipboard operations. This paper first outlines the maritime environment of the helicopter ship interface within ROSIE. Second, the paper investigates in a ship tracking model using 6D pose estimation. It then describes a novel visual augmentation concept using a see-through HMD to visualize the detected landing platform. Third, a pilot assistance system (PAS), consisting of the visual augmentation and two advanced flight control modes within the Automatic Flight Control System (AFCS) is designed together with two professional maritime test pilots. Lastly, the paper reports on a piloted simulation test of maritime mission task elements during the helicopter shipboard final recovery mission, in which the HMD visual augmentation concept is evaluated by two maritime test pilots from the German Armed Forces. Piloted evaluations identified a reduced pilot workload down to Level 1 while operating the helicopter with the visual augmentation concept. In parallel, observations on the flight performance indicated a more stable approach while operating the helicopter with the control augmentation being activated. Finally, the ship tracking model using 6D pose estimation shows promising results as to be used in the simulation environment to detect the ship within DVE conditions.