The excellent properties of ultra-fine grained (UFG) materials are relevant to substantial grain refinement and the corresponding induced small grains delineated by high-angle grain boundaries. The present study aims to understand the grain refinement mechanism by examining the nickel single crystal processed by high pressure torsion (HPT), a severe plastic deformation method to produce UFG materials based upon crystal plasticity finite element (CPFEM) simulations. The predicted grain maps by the developed CPFEM model are capable of capturing the prominent characteristics associated with grain refinement in HPT. The evolution of the orientation of structural elements and the rotations of crystal lattices during the HPT process of the detected differently oriented grains are extensively examined. It has been found that there are mainly two intrinsic origins of lattice rotation which cause the initial single crystal to subdivide. The correlation between the crystallographic orientation changes and lattice rotations with the grain fragmentation are analyzed and discussed in detail based on the theory of crystal plasticity.