Pulsed lasers operating in the mid-infrared (3–20 μm) are important for a wide range of applications in sensing, spectroscopy, imaging and communications. Despite recent advances with mid-infrared gain platforms, the lack of a capable pulse generation mechanism remains a significant technological challenge. Here we show that bulk Dirac fermions in molecular beam epitaxy grown crystalline Cd3As2, a three-dimensional topological Dirac semimetal, constitutes an exceptional ultrafast optical switching mechanism for the mid-infrared. Significantly, we show robust and effective tuning of the scattering channels of Dirac fermions via an element doping approach, where photocarrier relaxation times are found flexibly controlled over an order of magnitude (from 8 ps to 800 fs at 4.5 μm). Our findings reveal the strong impact of Cr doping on ultrafast optical properties in Cd3As2 and open up the long sought parameter space crucial for the development of compact and high-performance mid-infrared ultrafast sources.