This paper presents a novel approach for human-machine interface using visual-based communication. Scene analysis, object recognition and tracking, together with gesture detection and classification, are employed to design and implement a system for real time interaction. A new paradigm is proposed for efficient selection of hand gestures that can be used in other gesturebased environments aiming at interactive multimedia. An inexpensive web camera and readily available tools (such as an ordinary pen) have been used in this approach to render the system cost effective and suitable for general and home-based multimedia applications. Focus is placed on real time applications with a fast decision function for gesture classification, and a model-based approach for orientation detection. Experimental results demonstrate the efficacy of the proposed approach when tested using different performance criteria.