When a fraction of the light reflected by an external cavity re-enters the laser cavity, both the amplitude and the frequency of the lasing field can be modulated. This phenomenon is called the self-mixing effect (SME). A self-mixing laser diode (SM-LD) is a sensor using the SME. Usually, such LDs operate below the stability boundary where no relaxation oscillation happens. The boundary is determined by the operation condition including the injection current, optical feedback strength and external cavity length. This paper discovers the features of an SM-LD where the LD operates beyond the stability boundary, that is, near the relaxation oscillation (RO) status. We call the signals from such a SM-LD as RO-SM signals to differentiate them from the conventional SM signals reported in the literature. Firstly, simulations are made based on the well-known Lang and Kobayashi (L-K) equations. Then the experiments are conducted on different LDs to verify the simulation results. It shows that a RO-SM signal exhibits high frequency oscillation with its amplitude modulated by a slow time varying envelop which corresponds to the movement of the external target. The envelope has same fringe structure (half-wavelength displacement resolution) with the conventional SM signals. However, the amplitudes of the RO-SM signals are much higher compared to conventional SM signals. The results presented reveal that an SM-LD operating near the RO has potential for achieving sensing with improved sensitivity.