Interactive Process Automation based on lightweight object detection in manufacturing processes

Interactive Process Automation refers to the idea of supporting the interaction of humans in processes through physical objects. This is particularly promising for human/cobot collaboration tasks where the communication is fuzzy. A typical example is a picking and placing scenario. Here, a “picking area” can serve as a user interface, i.e., objects are freely placed in a defined area, and then identified and transferred to specific positions, where deterministic processes can use them. If, for example, object A is placed at position pos_A by the human, automatically, the robot is instructed to pick A and place it at position pos_B on a tray. Realizing Interactive Process Automation for picking and placing tasks in manufacturing processes requires (i) a lightweight and flexible object detection approach and (ii) a human–machine interface design for Interactive Process Automation. This work proposes (i) an object detection approach that works solely based on synthetic training data. The object detection is embedded into (ii) generic process models that are implemented based on an existing manufacturing orchestration framework and a camera-equipped cobot. The approach is prototypically implemented and evaluated based on several experiments including a pick and place cobot station.