Phase shifting profilometry can achieve high accuracy for the 3D shape measurement of static object. Errors will be introduced when the object is moved during the movement. The fundamental reason causing the above issue is: PSP requires multiple fringe patterns but the reconstruction model does not include the object movement information. This paper proposes a new method to automatically measure the 3D shape of the rigid object with arbitrary 2D movement. Firstly, the object movement is tracked by the SIFT algorithm and the rotation matrix and translation vector describing the movement are estimated. Then, with the reconstruction model including movement information, a least-square algorithm is applied to retrieve the correct phase value. The proposed method can significantly reduce the errors caused by the object movement. The whole reconstruction process does not need human intervention and the proposed method has high potential to be applied in industrial applications. Experiments are presented to verify the effectiveness.