This paper introduces a new paradigm for abnormal behavior detection relying on the integration of contextual information in Markov random fields. Contrary to traditional methods, the proposed technique models the local density of object feature vector, therefore leading to simple and elegant criterion for behavior classification. We develop a Gaussian Markov random field mixture catering for multi-modal density and integrating the neighborhood behavior into a local estimate. The convergence of the random field is ensured by online learning through a stochastic clustering algorithm. The system is tested on an extensive dataset (over 2800 vehicles) for behavior modeling. The experimental results show that abnormal behavior for a pedestrian walking, running and cycling on the highway, is detected with 82% accuracy at the 10% false alarm rate, and the system has an overall accuracy of 86% on the test data.