A three-dimensional (3D) magnetic property testing system has been completed and successfully used to measure 3D hysteresis loci of soft magnetic material. This paper presents the techniques to characterize soft magnetic materials under 3D magnetic excitations in detail. Using three couples of excitation coils controlled by a computer to generate the magnetic fields in three orthogonal axes, various types of flux density loci, such as circular or elliptical rotating vectors of flux density with any given orientations in 3D space, can be obtained. Based on 3D finite element analysis and a comparative study, a sandwich arrangement comprising a sample, guard pieces, and search coils with double-layer structure was proposed. Compared with the conventional surface search coils, this arrangement can significantly improve the accuracy of measurement. The comprehensive calibration process and the experimental results of the 3D hysteresis loci of soft magnetic composite materials are also presented.