Large datasets play a crucial role in the progression of surgical robotics, facilitating advancements in the fields of surgical task recognition and automation. Moreover, public datasets enable the comparative analysis of various algorithms and methodologies, thereby assessing their effectiveness and performance. The ROSMA (Robotics Surgical Maneuvers) dataset provides 206 trials of common surgical training tasks performed with the da Vinci Research Kit (dVRK). In this work, we extend ROSMA dataset with two annotated subsets: ROSMAT24, which contains bounding box annotations for instrument detection, and ROSMAG40, which contains high and low level gesture annotations. We propose an annotation method that provides independent labels for the right-handed tool and the left-handed tool. For instrument identification, we validate our proposal with a YOLOv4 model in two experimental scenarios. We demonstrate the generalization capabilities of the network to detect instruments in unseen scenarios. For gesture segmentation, we propose two label categories: high-level annotations that describe gestures at a maneuvers level, and low-level annotations that describe gestures at a fine-grain level. To validate this proposal, we have designed a recurrent neural network based on a bidirectional long-short term memory layer. We present results for four cross-validation experimental setups, reaching up to a 77.35% mAP.

Abstract: Hand gestures are an essential kind of nonverbal communication that let people engage and be expressive with one another. As technology has progressed, hand gesture recognition systems have become an important field of study, providing a link between human activities and artificial intelligence. An overview of the field of hand gesture recognition systems is given in this abstract, along with an explanation of its importance, difficulties, and possible uses. This abstract begins by examining the role that hand gestures have in communication. It then goes on to show the cultural significance and wide range of variations in gestural expressions that occur in many contexts and societies. Artificial intelligence systems face a unique problem in comprehending and interpreting these motions, which calls for the combination of computer vision, machine learning, and pattern recognition approaches. The abstract goes on to address why people develop hand motions. The purpose behind creating systems to recognize hand gestures is also covered in the abstract, with a focus on how these systems could improve human-computer interaction in several fields, including robotics, virtual reality, healthcare, and assistive technologies. These systems have the potential to enhance accessibility, efficiency, and user experience by allowing people to engage with devices and interfaces naturally and intuitively. The abstract also discusses environmental elements including backdrop clutter and lighting, as well as the technological difficulties in identifying and understanding hand gestures, which include variations in hand forms, orientations, and movements. To tackle these obstacles, strong algorithms that can record and interpret complex hand movements in real-time must be developed.

Gesture recognition has a crucial role in Human-Computer Interaction (HCI) and in assisting the elderly to perform automatically their everyday activities. In this paper three methods for gesture recognition and computer vision were implemented and tested in order to investigate the most suitable one. All methods, Machine learning using IMU, Machine learning on device and were combined with certain activities that were determined during a needs analysis research. The same volunteers took part to the pilot testing of the proposed methods. The results highlight the strengths and weaknesses of each approach, revealing that 60 while some methods excel in specific scenarios, the integrated solution of MoveNet and 61 CNN provides a robust framework for real-time gesture recognition.

We investigated preschool-aged children’s understanding of early fractional tasks and how that performance correlates with fine motor skills and use of gestures while counting. Participants were 33 preschoolers aged 4 to 5 in two Southeastern public elementary schools. Children were tested individually in an interview-like setting. Mathematics tasks were presented in a paper and pencil format and the Grooved Pegboard test assessed fine motor skills. Finally, utilization of gestures was evaluated by taking a behavioral rating of the child’s hand morphology, accuracy of gestures, and synchrony of gestures and spoken word while performing a counting task. Results indicate performance on fractional reasoning tasks significantly predicts both fine motor ability and accuracy of gestures.

To achieve an acceptable level of security on the web, the Completely Automatic Public Turing test to tell Computer and Human Apart (CAPTCHA) was introduced as a tool to prevent bots from doing destructive actions such as downloading or signing up. Mobile devices have small screens, and therefore, using the common CAPTCHA methods (e.g. text CAPTCHAs) in these devices raises usability issues. To introduce a reliable, secure, and usable CAPTCHA that is suitable for mobile devices, this paper introduces a hand gesture recognition CAPTCHA based on applying Genetic Algorithm (GA) principles on Multi-Layer Perceptron (MLP). The proposed method improves the performance of MLP-based hand gesture recognition. It has been trained and evaluated on 2201 videos of the IPN Hand dataset, and MSE and RMSE benchmarks report the index values of 0.0018 and 0.0424, respectively. Comparison with the related works shows a minimum of 1.79% fewer errors, and experiments produced a sensitivity of 93.42% and accuracy of 92.27% – 10.25% and 6.65% improvement compared to the MLP implementation.

This paper introduces a multi-class hand gesture recognition model developed to identify a set of defined hand gesture sequences in two-dimensional RGB video recordings. The work presents an action detection classifier that looks at both appearance and spatiotemporal parameters of consecutive frames. The classifier utilizes a convolutional-based network combined with a long-short-term memory unit. To leverage the need for a large-scale dataset, the model uses an available dataset to then adopt a technique known as transfer learning to fine-tune the model on the hand gestures of relevance. Validation curves performed over a batch size of 64 indicate an accuracy of 93.95% (± 0.37) with a mean Jaccard index of 0.812 (± 0.105) for 22 participants. The presented model illustrates the possibility of training a model with a small set of data (113,410 fully labelled frames). The proposed pipeline embraces a small-sized architecture that could facilitate its adoption.

Wearable technology has been proposed as a potential tool to change the way of human life, such as the smart bracelet and the Google Glass. In the wearable technology, the inertial sensor has great significance in tracking the object movements. The paper focused on detecting the movements of user’s finger based on the inertial sensor to give the control signals. Firstly, the attitude matrix, which represented the transformation relation of carrier coordinate system and the navigation coordinate system, was obtained. Secondly, the attitude matrix was expressed based on the quaternions. Thirdly, the finger gesture was processed by the attitude matrix to get the attitude angle. Finally, the robot was controlled by attitude angle to make the moving action. The experimental results showed the detection of the finger movement is effective.

The surface electromyogram (sEMG)--based gesture recognition to the development of intelligent prostheses has emerged as a promising avenue for upper limb amputees. However, the temporal variations in sEMG have rendered recognition models less efficient than anticipated. Cross-session calibration and training data indicate the capacity to reduce this variation. The impact of varying calibration and training data amounts on gesture recognition performance for amputees is still unknown. To assess these effects, we present four datasets for the evaluation of calibration data and examine the impact of training data volume on benchmark performance. Two amputees who had undergone amputations years prior were recruited and seven sessions of data were collected for analysis from each of them. The experimental results show that the calibration data improved the average accuracy per subject by 3.03% and 6.16%, respectively, compared to the baseline results. Moreover, it was found that to collect training data for modeling, boosting the number of training sessions proves more efficient than increasing the number of trials within a single session. Two potentially effective strategies are proposed in light of this research to enhance the cross-session models further. We consider these findings to be of the utmost importance for the commercialization of intelligent prostheses, as they demonstrate the criticality of gathering calibration and cross-session training data, as well as offer effective strategies for maximizing the utilization of the entire dataset.

In this work, we show that a late fusion approach to multi-modality in sign language recognition improves the overall ability of the model in comparison to the singular approaches of Computer Vision (88.14%) and Leap Motion data classification (72.73%). With a large synchronous dataset of 18 BSL gestures collected from multiple subjects, two deep neural networks are benchmarked and compared to derive a best topology for each. The Vision model is implemented by a CNN and optimised MLP and the Leap Motion model is implemented by an evolutionary optimised deep MLP topology search. Next, the two best networks are fused for synchronised processing which results in a better overall result (94.44%) since complementary features are learnt in addition to the original task. The hypothesis is further supported by application of the three models to a set of completely unseen data where a multi-modality approach achieves the best results relative to the single sensor method. When transfer learning with the weights trained via BSL, all three models outperform standard random weight distribution when classifying ASL, and the best model overall for ASL classification was the transfer learning multi-modality approach which scored 82.55% accuracy.

With the development of artificial intelligence technology, manipulators, as an important part of the work of intelligent robots, can replace humans to perform various complex tasks. In recent years, there have been many studies on the remote control of manipulators using various control technologies. Starting from the manufacturing materials of flexible sensors used for manipulators, this paper introduces the base materials, sensing materials, and flexible electrode materials of flexible sensors, respectively, and summarizes the performance of flexible sensors made based on the characteristics of different materials. The basic principles of dipping and coating, lithography, inkjet printing, screen printing, 3D printing and so on are introduced from the perspective of flexible sensor manufacturing process. Some different functions of sensors achieved by different structural designs, such as: tensile cracking, microchannels, etc. are also introduced. Then, from the perspective of recognition accuracy of the manipulator, two kinds of control methods based on flexible sensor by data glove control and surface EMG control are classified and introduced, and the problems existing in three aspects of flexible sensor material preparation, manufacturing process and control series design are summarized. Finally, the possible future research directions in this field are suggested.

The paper presents the possibility of using KINECT v2 module to control an industrial robot by means of gestures and voice commands. It describes elements of creating software for off-line and on-line robot control. The application for KINECT module was developed in C# language in Visual Studio environment, while the industrial robot control program was developed in RAPID language in RobotStudio environment. The development of a two-threaded application in RAPID language allowed to separate two independent tasks for the IRB120 robot. The main task of the robot is performed in thread no. 1 (responsible for movement). Simultaneously working thread no. 2 ensures continuous communication with the KINECT system and provides information about the gesture and voice commands in real time without any interference in thread no. 1. The applied solution allows the robot to work in industrial conditions without negative impact of communication task on the time of robot’s work cycles. Thanks to the development of a digital twin of the real robot station, tests of proper application functioning in off-line mode (without using a real robot) were conducted. Obtained results were verified online (on the real test station). Tests of correctness of gesture recognition were carried out, the robot recognized all programmed gestures. Another test carried out was the recognition and execution of voice commands. A difference in the time of task completion between the actual and virtual station was noticed - the average difference was 0.67 s. The last test carried out was to examine the impact of interference on the recognition of voice commands. With a 10dB difference between the command and noise, the recognition of voice commands was equal to 91.43%. The developed computer programs have a modular structure, which enables easy adaptation to process requirements.

Research on bidirectional human-machine interfaces will enable the smooth interaction with robotic platforms in contexts ranging from industry to tele-medicine and rescue. This paper introduces a bidirectional communication system to achieve multisensory telepresence during the gestural control of an industrial robotic arm. We complement the gesture-based control by means of a tactile-feedback strategy grounding on a spiking artificial neuron model. Force and motion from the robot are converted in neuromorphic haptic stimuli delivered on the user’s hand through a vibro-tactile glove. Untrained personnel participated in an experimental task benchmarking a pick-and-place operation. The robot end-effector was used to sequentially press six buttons, illuminated according to a random sequence, and comparing the tasks executed without and with tactile feedback. The results demonstrated the reliability of the hand tracking strategy developed for controlling the robotic arm, and the effectiveness of a neuronal spiking model for encoding hand displacement and exerted forces in order to promote a fluid embodiment of the haptic interface and control strategy. The main contribution of this paper is in presenting a robotic arm under gesture-based remote control with multisensory telepresence, demonstrating for the first time that a spiking haptic interface can be used to effectively deliver on the skin surface a sequence of stimuli emulating the neural code of the mechanoreceptors beneath.

This paper presents a system developed for the assistance with a collaborative robot in hand-assisted laparoscopic surgery (HALS). The system includes a sensing glove with piezoresistive sensors which capture continuously the flexion degree of the surgeon's fingers. These data are analyzed using an algorithm that detects and recognize the selected movements. This information is sent as commands to the collaborative robot throughout the surgical operation. The bending patterns, speed and execution times of the movements are modelled in a pre-phase in which it will extract all the necessary information for later detection during the motion execution. The results obtained with 10 different volunteers show a high degree of accuracy and a low false discovery rate.

In order to improve real-time performance of gesture recognition by micro-Doppler map of mmWave Radar, the point cloud based gesture recognition for mmWave Radar is proposed in this paper.Two steps are carried out for mmWave Radar based gesture recognition. The first step is to estimate the point cloud of the gestures by 3D-FFT and the peak grouping. The second step is to train the TRANS-CNN model by combining the multi-head self-attention and the 1D-convolutional network, so as to extract the features in the point cloud data at a deeper level to categorize the gestures. In the experiments, TI mmWave Radar sensor IWR1642 is used as benchmark to evaluate the feasibility of the proposed approach. The results show that the accuracy of the gesture recognition reaches 98.5%. In order to prove the effectiveness of our approach, a simply 2Tx2Rx Radar sensor is developed in our lab and the accuracy of recognition reaches 97.1%. The results show that our proposed gesture recognition approach achieves the best performance in real-time with limited training data, in comparison with the existing methods.

The rapid growth of smartphone financial services raises the need for secure mobile authentication. Continuous authentication is a user-friendly way to strengthen the security of smartphones by implicitly monitoring a user’s identity through sessions. Mobile continuous authentication can be viewed as an anomaly detection problem in which models discriminate between one genuine user and the rest of the imposters (anomalies). In practice, complete imposter profiles are hardly available due to the time and monetary cost, while leveraging genuine data alone yields poor generalized models due to the lack of knowledge about imposters. To address this challenge, we recast continuous mobile authentication as a few-shot anomaly detection problem, aiming to enhance the inference robustness of unseen imposters by using partial knowledge of available imposter profiles. Specifically, we propose a novel deep learning-based model, namely Local Feature Pooling based Temporal Convolution Network (LFP-TCN), which directly models raw sequential mobile data, aggregating global and local feature information. In addition, we introduce a random pattern mixing augmentation to generate class-unconstrained imposter data for the training. The augmented pool enables characterizing various imposter patterns from limited imposter data. Finally, we demonstrate practical continuous authentication using score-level fusion, which prevents long-term dependency or increased model complexity due to extended re-authentication time. Experiments on two public benchmark datasets show the effectiveness of our method and its state-of-the-art performance.

Gesture recognition has always been a technique to decrease the distance between the physical and the digital world. In this work, we introduce an Arduino based vehicle system which no longer require manual controlling of the cars. The proposed work is achieved by utilizing the Arduino microcontroller, accelerometer, RF sender/receiver, and Bluetooth. Two main contributions are presented in this work. Firstly, we show that the car can be controlled with hand-gestures according to the movement and position of the hand. Secondly, the proposed car system is further extended to be controlled by an android based mobile application having different modes (e.g., touch buttons mode, voice recognition mode). In addition, an automatic obstacle detection system is introduced to improve the safety measurements to avoid any hazards. The proposed systems are designed at lab-scale prototype to experimentally validate the efficiency, accuracy, and affordability of the systems. We remark that the proposed systems can be implemented under real conditions at large-scale in the future that will be useful in automobiles and robotics applications.

A tabletop system can facilitate multi-user collaboration in a variety of settings including small meetings, group work, and education and training exercises. The ability of identifying the users touching the table and their positions can promote collaborative work among participants, so methods have been studied that involve the attaching of sensors to the table or chairs or to the users themselves. An effective method of recognizing user actions without placing a burden on the user would be some type of visual process, so the development of a method that processes multi-touch gestures by visual means is desired. This paper describes the development of a multi-touch tabletop system using infrared image recognition for user position identification and presents the results of touch-gesture recognition experiments and a system usability evaluation. Using an FTIR touch panel and infrared light, this system picks up the shadow area of the user’s hand by infrared camera in relation to user touch operations and estimates user position by image recognition. The multi-touch gestures prepared for this system include an operation to change the direction of an object to face the user and a copy operation in which two users generate duplicates of an object. The average recognition rate of the change-direction gesture and copy gesture were found to be 96% and 85%, respectively. In addition, the system usability evaluation revealed that prior learning was easy and that system operations could be easily performed.

Gesture recognition technology has been quickly developed in the field of human-computer interaction. The multiple-input multiple-output (MIMO) radar has been widely adopted in gesture recognition because of its notable spatial resolution. In this work, a highly accurate MIMO radar-based hand gesture recognition algorithm with very low complexity is proposed. To make the proposed system applicable in the industry and work well even when the size of the training data is limited, we applied several low-complexity adaptive signal processing methods to extract features and reduce the noise effect. First, spectrum analysis is applied to range-Doppler maps (RDMs) and a cell-averaging constant false alarm rate (CA-CFAR) with mirror filters is applied to improve the robustness to noise. Afterward, the features related to the distance, speed, direction, and the elevation angle of the moving object are determined by the proposed adaptive signal analysis techniques and the random forest is applied for classification. The proposed system has the capability to precisely distinguish and identify eight motions, including waving, moving to the left or right, patting, pushing, pulling, and rotating clockwise or anticlockwise, with an accuracy of 95%. Experiments demonstrate the capability of the proposed hand gesture recognition system to classify different movements precisely.

Hand gestures are an essential part of human-to-human communication and interaction, and therefore for technical applications, the aim is increasingly to achieve interaction between humans and computers that is as natural as possible, for example by means of natural language or hand gestures. In the context of human-machine interaction research, these methods are consequently being explored more and more. However, the realization of natural communication between humans and computers is a major challenge. In the field of hand gesture recognition, research approaches are being pursued that use additional hardware such as special gloves to classify gestures with high accuracy. Recently, deep learning techniques using artificial neural networks have been increasingly proposed for the problem of gesture recognition without using such tools. In this context, in our approach convolutional neural network (CNN) will be explored in detail for the task of hand gesture recognition. CNN is a deep neural network that can be used in the field of visual object processing and classification. The goal of this work is to recognize ten types of static hand gestures in front of complex backgrounds and different hand sizes based on raw images without the use of extra hardware. We achieved good results with a CNN network architecture consisting of seven layers. Through data augmentation and skin segmentation, a significant increase of the model accuracy was achieved. On public benchmarks the ten gestures have been classified almost perfectly with a testing accuracy of 96.5%

Leader-follower autonomy (LFA) systems have so far only focused on vehicles following other vehicles. Though there have been several decades of research into this topic, there has not yet been any work on human-vehicle leader-follower systems. We present a system in which an autonomous vehicle — our ACTor 1 platform — can follow a human leader who controls the vehicle through gestures. We successfully developed a modular pipeline that uses artificial intelligence/deep learning to recognize hand-and-body gestures from a user in view of the vehicle’s camera and translate those gestures into physical action by the vehicle. We demonstrate our work using our ACTor 1 platform, a modified Polaris Gem 2 electric vehicle. This work has numerous applications, such as material transport in industrial contexts.

Hand gestures may play an important role in medical applications for health care of elderly people, where providing a natural interaction for different requests can be executed by making specific gestures. In this study we explored three different scenarios using a Microsoft Kinect V2 depth sensor then evaluated the effectiveness of the outcomes. The first scenario utilized the default system embedded in the Kinect V2 sensor, which depth metadata gives 11 parameters related to the tracked body with five gestures for each hand. The second scenario used joint tracking provided by Kinect depth metadata and depth threshold together to enhance hand segmentation and efficiently recognize the number of fingers extended. The third scenario used a simple convolutional neural network with joint tracking by depth metadata to recognize five categories of gestures. In this study, deaf-mute elderly people execute five different hand gestures to indicate a specific request, such as needing water, meal, toilet, help and medicine. Then, the requests were sent to the care provider’s smartphone because elderly people could not execute any activity independently. The system transferred these requests as a message through the global system for mobile communication (GSM) using a microcontroller.

Due to Motor Neurone Diseases, a large population remains disabled worldwide, negatively impacting their independence and quality of life. This typically involves a weakness in the hand and forearm muscles, making it difficult to perform fine motor tasks such as writing, buttoning a shirt, or gripping objects. This project presents a vision-enabled rehabilitation hand exoskeleton to assist disabled persons in their hand movements. The design goal was to create an accessible tool to help with a simple interface requiring no training. This prototype is built on a commercially available glove where a camera and embedded processor were integrated to help open and close the hand, using air pressure, thus grabbing an object. An accelerometer is also implemented to detect the characteristic hand gesture to release the object when desired. This passive vision-based control differs from active EMG-based designs as it does not require individualized training. Continuing the research will reduce the cost, weight, and power consumption to facilitate mass implementation.