Many shape memory alloy (SMA)-based soft actuators have specific composite structures and manufacture processes, and are therefore unique. However, these exclusive characteristics limit their capabilities and applications, so in this article a soft and smart digital structure (SDS) is proposed that acts like a modular unit to assemble soft actuators by a layered adhesive bonding process. The SDS is a fully soft structure that encapsulates a digital skeleton consisting of four groups of parallel and independently actuated SMA wires capable of outputting a four-channel tunable force. The layered adhesive bonding process modularly bonds several SDSs with an elastic backbone to fabricate a layered soft actuator where the elastic backbone is used to recover the SDSs in a cooling process using the SMA wires. Two kinds of SDS-based soft actuators were modularly assembled, an actuator, SDS-I, with a two-dimensional reciprocal motion, and an actuator, SDS-II, capable of bi-directional reciprocal motion. The thermodynamics and phase transformation modeling of the SDS-based actuator were analyzed. Several extensional soft actuators were also assembled by bonding the SDS with an anomalous elastic backbone or modularly assembling the SDS-Is and SDS-IIs. These modularly assembled soft actuators delivered more output channels and a complicated motion, e.g., an actinomorphic soft actuator with four SDS-Is jumps in a series of hierarchical heights and directional movement by tuning the input channels of the SDSs. This result showed that the SDS can modularly assemble multifarious soft actuators with diverse capabilities, steerability and tunable outputs.