In this paper, we demonstrate how texture classification and retrieval could benefit from learning perceptual pairwise distance of different texture classes. Textures as represented by certain image features may not be correctly compared in a way that is consistent with human perception. Learning similarity helps to alleviate this perceptual inconsistency. For textures, psychological experiments were shown to be able to construct perceptual pairwise distance matrix. We are going to show how this distance information could be utilized in learning similarity by Support Vector Machines for efficient texture classification and retrieval.