Accurate prostate localization is the key to the success of radiotherapy. It remains a difficult problem for CT images due to the low image contrast, the prostate motion, and the uncertain presence of rectum gas. In this paper, a learning based framework is proposed to improve the accuracy of prostate detection in CT. It adaptively determines distinctive feature types at distinctive image regions, thus filtering out features that are salient in image appearance, but irrelevant to prostate localization. Furthermore, an image similarity function is learned to make the image appearance distance consistent with the underlying prostate alignment. The efficacy of our proposed method has been demonstrated by the experiment.