Midrange Geometric Interactions for Semantic Segmentation: Constraints for Continuous Multi-label Optimization

In this article we introduce the concept of midrange geometric constraints into semantic segmentation. We call these constraints ‘midrange’ since they are neither global constraints, which take into account all pixels without any spatial limitation, nor are they local constraints, which only regard single pixels or pairwise relations. Instead, the proposed constraints allow to discourage the occurrence of labels in the vicinity of each other, e.g., ‘wolf’ and ‘sheep’. ‘Vicinity’ encompasses spatial distance as well as specific spatial directions simultaneously, e.g., ‘plates’ are found directly above ‘tables’, but do not fly over them. It is up to the user to specifically define the spatial extent of the constraint between each two labels. Such constraints are not only interesting for scene segmentation, but also for part-based articulated or rigid objects. The reason is that object parts such as for example arms, torso and legs usually obey specific spatial rules, which are among the few things that remain valid for articulated objects over many images and which can be expressed in terms of the proposed midrange constraints, i.e. closeness and/or direction. We show, how midrange geometric constraints are formulated within a continuous multi-label optimization framework, and we give a convex relaxation, which allows us to find globally optimal solutions of the relaxed problem independent of the initialization.