Copyright 2020 Honson, Huynh-Thu, Arnison, Monaghan, Isherwood and Kim. This study examined perceptual differentiation of specular from diffuse shading for the recovery of surface color and gloss. In Experiment 1, we parametrically varied the mesoscale relief height of globally planar surfaces, specular sharpness and the orientation of the surface relative to the light source. We obtained psychophysical matches for perceived color saturation and value (HSV), but also considered whether the main effects could be influenced by color space used when transforming data to perceptually-uniform CIE LCH space. Results revealed strong interactions between perceived color attributes and the lighting conditions, the structure of specular reflections, and surface relief. Declines in saturation were observed with increasing specular roughness (using an HSV color representation), but no similar decline was observed in chroma (using a CIE LCH color representation). Experiment 2 found strong negative correlations between perceived gloss and specular roughness. Perceived gloss also depended on mesoscopic relief height and orientation of the surface relative to the light source. Declines in perceived gloss moderately accounted for the variability in color saturation and value matches obtained in Experiment 1. We found information about perceived specular coverage could further improve the model's accountability of perceived color saturation and lightness (Experiment 3). These findings together suggest that perceived color saturation and color value depends on the visual system's ability to distinguish the underlying diffuse shading from specular highlights in images.