Series Elastic Actuators (SEAs) have several mechanical superiorities over conventional stiff and non-back-drivable actuators, e.g., lower reflected inertia at output, greater shock tolerance, low cost force measurement, energy storage, safety, and so on. However, their applications generally suffer from performance limitations, particularly in position control, due to insufficient controller designs. This paper proposes a unified Active Disturbance Rejection (ADR) motion controller for the robust position and force control problems of SEAs by using Differential Flatness (DF) and Disturbance Observer (DOb). It can suppress not only matched but also mismatched disturbances. Robust state and control input references are systematically generated in terms of a fictitious design variable, namely differentially flat output, estimations of disturbances, and their successive time derivatives. The proposed robust motion controller improves the performance of SEAs when they suffer from internal and external disturbances, such as friction, inertia variation and external load, in real implementations. Experimental results are given to validate the proposal.