An approach to color image understanding is presented that can be used to segment and analyze surfaces with color variations due to highlights and shading. A theory--the dichromatic reflection model--that relates the reflected light from dielectric materials, such as plastic, to fundamental physical reflection processes, and describes the color of the reflected light as a linear combination of the color of the light due to surface reflection (highlights) and body reflection (object color) is discussed. This dichromatic theory is used in an algorithm that separates a color image into two parts: an image of just the highlights, and the original image with the highlights removed. It is shown how to perform automatic segmentation by applying this theory in stages to identify the object and highlight colors. The result is a combination of segmentation and reflection analysis that is better than traditional heuristic segmentation methods and provides important physical information about the surface geometry and material properties at the same time.