Towards Trustworthy AI: Sandboxing AI-Based Unverified Controllers for Safe and Secure Cyber-Physical Systems

In the past decade, artificial-intelligence-based (AI-based) techniques have been widely applied to design controllers over cyber-physical systems (CPSs) for complex control missions (e.g., motion planning in robotics). Nevertheless, AI-based controllers, particularly those developed based on deep neural networks, are typically very complex and are challenging to be formally verified. To cope with this issue, we propose a secure-by-construction architecture, namely Safe-Sec-visor architecture, to sandbox AI-based unverified controllers. By applying this architecture, the overall safety and security of CPSs can be ensured simultaneously, while formal verification over the AI-based controllers is not required. Here, we consider invariance and opacity properties as the desired safety and security properties, respectively. Accordingly, by leveraging a notion of (augmented) control barrier functions, we design a supervisor to check the control inputs provided by the AI-based controller and decide whether to accept them. At the same time, a safety-security advisor runs in parallel and provides fallback control inputs whenever the AI-based controller is rejected for safety and security reasons. To show the effectiveness of our approaches, we apply them to a case study on a quadrotor controlled by an AI-based controller. Here, the initial state of the quadrotor contains secret information which should not be revealed while the safety of the quadrotor should be ensured.