An integrated approach to shape and color-based image retrieval of rotated objects using hidden Markov models

An integrated approach to shape and color-based image retrieval, where the cues color and shape are both utilized in a local rather than a global way, is presented in this paper. An experimental retrieval system has been developed, which enables the user to search a color image database intuitively by presenting simple sketches. In order to be able to perform an elastic matching, which is especially needed in sketch-based image retrieval, objects in the images are represented by Hidden Markov Models. The use of streams (sets of features that are assumed to be statistically independent) within the HMM framework allows the integration of shape and color derived features into a single model, thereby allowing to control the influence of the different streams via stream weights. The approach has been evaluated on a color image database containing 120 different isolated objects with arbitrary orientation and showed good retrieval results with several users. Furthermore, the use of HMMs allows efficient pruning and thus a fast retrieval even with large databases.