ToF/Radar early feature-based fusion system for human detection and tracking

Industry 4.0 has become a general keyword over the last years. It is based on the inclusion of automation by increasing connectivity in various tasks during the production process. This fact did not exclude the human's effort whose presence remains important, especially the interaction between humans and robots will be a key element in the future manufacturing. In automated production lines, we find both humans and robots operating side-by-side in hybrid workplaces. The major focus for this workplaces today and in the future is to establish a safe work environment. However, what if safety meets "collaborative efficiency"? The system presented in this paper relies on the fusion of data coming from a Time of Flight (ToF) sensor and a 60 GHz radar sensor. The data are analyzed and evaluated using deep learning (DL) algorithms. The purpose is to detect humans and track their movements in the observed area. The resulted perception system can be installed somewhere in a room or on a moving system. A first demonstrator has been developed, tested and evaluated. An additional graphical interface was developed to show in real time the capability of the data fusion system. The system can detect up to 5 persons in a selected area with 98% confidentiality. The so-described system is able as well to estimate each person DoM and the person's instantaneous speed and position. Based on the output of our developed system, it is possible to define industrial use cases as well as many other different applications in different fields.