Navigation and manipulation planning using a visuo-haptic sensor on a mobile platform

Mobile systems interacting with objects in unstructured environments require both haptic and visual sensors to acquire sufficient scene knowledge for tasks such as navigation and manipulation. Typically, separate sensors and processing systems are used for the two modalities. We propose to acquire haptic and visual measurements simultaneously, providing naturally coherent data. For this, compression of a passive, deformable foam rod mounted on the actuator is measured visually by a low-cost camera, yielding a 1-D stress function sampled along the contour of the rod. The same camera observes the nearby scene to detect objects and their reactions to manipulation. The system is passively compliant and the complexity of the sensor subsystems is reduced. Furthermore, we present an integrated approach for navigation and manipulation on mobile platforms, which integrates haptic data from the sensor. A high-level planning graph represents both the structure of a visually acquired map, as well as manipulable obstacles. Paths within this graph represent high-level navigation and manipulation tasks, e.g., pushing of obstacles. A cost-optimal task plan is generated using standard pathfinding techniques. The approach is implemented and validated on a mobile robotic platform. Obtained forces are compared with a reference, showing high accuracy within the medium sensor range. A real-world experiment is presented, which uses the sensor for haptic exploration of obstacles in an office environment. Substantially faster task plans can be found in cluttered scenes compared with purely visual navigation.