ToF/Radar early feature-based fusion system for human detection and tracking

Feryel Zoghlami, Okan Kamil Sen, Harald Heinrich, Germar Schneider, Emec Ercelik, Alois Knoll, Thomas Villmann

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

8 Scopus citations


Industry 4.0 has become a general keyword over the last years. It is based on the inclusion of automation by increasing connectivity in various tasks during the production process. This fact did not exclude the human's effort whose presence remains important, especially the interaction between humans and robots will be a key element in the future manufacturing. In automated production lines, we find both humans and robots operating side-by-side in hybrid workplaces. The major focus for this workplaces today and in the future is to establish a safe work environment. However, what if safety meets "collaborative efficiency"? The system presented in this paper relies on the fusion of data coming from a Time of Flight (ToF) sensor and a 60 GHz radar sensor. The data are analyzed and evaluated using deep learning (DL) algorithms. The purpose is to detect humans and track their movements in the observed area. The resulted perception system can be installed somewhere in a room or on a moving system. A first demonstrator has been developed, tested and evaluated. An additional graphical interface was developed to show in real time the capability of the data fusion system. The system can detect up to 5 persons in a selected area with 98% confidentiality. The so-described system is able as well to estimate each person DoM and the person's instantaneous speed and position. Based on the output of our developed system, it is possible to define industrial use cases as well as many other different applications in different fields.

Original languageEnglish
Title of host publicationProceedings - 2021 22nd IEEE International Conference on Industrial Technology, ICIT 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages8
ISBN (Electronic)9781728157306
StatePublished - 10 Mar 2021
Event22nd IEEE International Conference on Industrial Technology, ICIT 2021 - Valencia, Spain
Duration: 10 Mar 202112 Mar 2021

Publication series

NameProceedings of the IEEE International Conference on Industrial Technology


Conference22nd IEEE International Conference on Industrial Technology, ICIT 2021


  • automated fabrication
  • deep learning
  • human/robotic collaboration
  • industry 4.0
  • machine learning
  • radar sensor
  • sensor fusion
  • time of flight camera


Dive into the research topics of 'ToF/Radar early feature-based fusion system for human detection and tracking'. Together they form a unique fingerprint.

Cite this