Wearable technology has been proposed as a potential tool to change the way of human life, such as the smart bracelet and the Google Glass. In the wearable technology, the inertial sensor has great significance in tracking the object movements. The paper focused on detecting the movements of user’s finger based on the inertial sensor to give the control signals. Firstly, the attitude matrix, which represented the transformation relation of carrier coordinate system and the navigation coordinate system, was obtained. Secondly, the attitude matrix was expressed based on the quaternions. Thirdly, the finger gesture was processed by the attitude matrix to get the attitude angle. Finally, the robot was controlled by attitude angle to make the moving action. The experimental results showed the detection of the finger movement is effective.