Optical feedback Self-mixing Interferometry (OFSMI) can achieve a high-resolution displacement sensing and measurement by using advanced digital signal processing. However, most existing signal processing algorithms used for OFSMI signals are implemented on a PC by Matlab or other programming languages. In this case, the whole structure of OFSMI sensing system is incompact and the measurement is in low speed. The design trends in sensing systems are towarding to small size, high integration and fast real time processing. These trends require us to improve the existing OFSMI design. It is a good solution to apply Field-programmable gate arrays (FPGAs) technique onto OFSMI sensing systems. In this work, we designed a FPGA based signal processing unit for an OFSMI displacement sensing system. The OFSMI sensing signals observed from an OFSMI system is connected to a FPGA development board (Spartan-3E) for high speed signal processing. The FPGA processing unit retrieves the displacement information carried in the OFSMI signals. The FPGA design includes noise reduction, signal peak detection and impulse magnitude tracking. As the magnitude of the sensing signal is time-varying, for adapting the variation, a dynamic updating algorithm is introduced in the magnitude tracking unit. Both simulation and hardware co-simulation show that the OFSMI system with a FPGA based signal processing unit can achieve fast and reliable displacement sensing.