This chapter first presents a brief review of the recent development in human action recognition. In particular, the principle and shortcomings of the conventional Hidden Markov Model (HMM) and its variants are discussed. We then introduce an expandable graphical model that represents the dynamics of human actions using a weighted directed graph, referred to as action graph. Unlike the conventional HMM, the action graph is shared by all actions to be recognized with each action being encoded in one or multiple paths and, thus, can be effectively and efficiently trained from a small number of samples. Furthermore, the action graph is expandable to incorporate new actions without being retrained and compromised. To verify the performance of the proposed expandable graphic model, a system that learns and recognizes human actions from sequences of silhouettes is developed and promising results are obtained.