Introduction

Literature Review

Methodology

3.1. Model Framework

This chapter focuses on discussing the outline of the system to achieve the objectives according to Chapter 1. Theoretically, the proposed method will be based accordingly with the selected literature review based on Chapter 2 using CNN algorithm-based model. The overall system data flow is represented in the below figures:

Figure 3.1 shows the overview of the overall system in a use case where a sign language is being detected and recognized by the system. The sign language recognition system depicts the concept of the system in recognizing the sign language by comparing the sign language that is being detected with the trained dataset of the system. Container-based virtualization (CBV) and hypervisor-based virtualization are among the most well-known virtualization of modern technology (Riskhan & Muhammad, 2017). More Container-based virtualization is depicted in Figure 3-2, which shows the system in a state diagram form.

Figure 3.1. Proposed Model Concept.

Figure 3.1. Proposed Model Concept.

Figure 3.2. Proposed System State Diagram.

Figure 3.2. Proposed System State Diagram.

The diagram above illustrates the flow of the system and its internal states. The key processes that are highlighted include the detection, recognition, classification, and translation processes. The detection process involves capturing the sign language via the device’s camera, followed by recognition and classification by the embedded system’s model, which has been trained using a dataset. It is important to note that the dataset itself is not stored within the system environment, but rather the trained model that was created using the dataset. The following section will provide a more in-depth explanation of these processes.

At this current stage, a gesture recognition system is proposed using MobileNet models. Chapter 2 shows that the MobileNet model is suitable to be implemented in this project, considering its efficient performance and its feasibility to be implemented in multiple devices. MobileNet is a convolutional neural network based that uses depth-wise separable convolutions to build light weight deep neural networks with low computational requirements (Howard et al., 2017). The use of CNN is advantageous in this project due to its ability in creating internal representation of a two-dimensional image that makes the model learn the input image efficiently. In other words, CNN is advantageous for spatial purposes such as image recognition. MobileNets is commonly used and provided in the TensorFlow library. TensorFlow is an end-to-end open source platform for machine learning and deep learning (Goldsborough, 2016).

3.2. Project Development

This chapter focuses on discussing the outline of the system to achieve the objectives according to Chapter 1. Theoretically, the proposed method will be based accordingly with the selected literature review based on Chapter 2 using CNN algorithm-based model. The overall system data flow is represented in the below figures:

3.2.1. Malaysian Sign Language Dataset

This project requires a dataset that consists of sign language images with the annotations of the images. The dataset consists of 330 images that are divided into five classes. The classes are divided based on the meaning of the sign. In this project, the classes are “hello”, “yes”, “no”, “thanks”, and “sorry” which represent the meaning of sign language. The images that are used are the pictures of the sign language gesture that demonstrate and depict the sign language that refers to the Malaysian sign language. The images of the datasets were extracted from the videos that were previously recorded by the author using the author as the object demonstration. The extracted images are then labeled using Label Studio in order to create the annotations. As a point of clearance, although the sign is classified and translated into English, the sign language still refers to the Bahasa Isyarat Malaysia (Official Malaysian Sign Language). The reference of the sign language that is used is Bahasa Isyarat Malaysia from Pocket Handbook Malaysian Sign Language by Malaysian Federation of the Deaf.

Figure 3.3 shows the collaged images that are extracted and labeled from the recorded videos that the author took. Besides the images, the dataset also contains the annotation files in the same number as the number of images. Inside the annotation files contain the information about the region or the bounding box that locates the object that the model is supposed to take as the training input. The bounding boxes are annotated using x and y coordinates within the picture. Figure 3.4 depicts the inside of an annotation file of “yes” class.

Figure 3.3. Malaysian Sign Language Dataset Images.

Figure 3.3. Malaysian Sign Language Dataset Images.

Figure 3.4. Dataset Annotation.

Figure 3.4. Dataset Annotation.

3.2.2. Sign Language Recognition Model

This section provides the description of the sign language recognition model that is used which in this project, MobileNetV1 model is used. This section covers the explanation about pretrained model and its architecture, as well as the related component such as the deployment model and its API.

3.2.2.1. MobileNetV1 Pretrained Model

The project focuses on using MobileNetV1 algorithm. To be specific, this project used a MobileNetV1 pretrained model that is embedded with a Single Shot Detector (SSD) and Feature Pyramid Network (FPN) from Tensorflow library. SSD is a single convolutional neural network object detection algorithm that is used to predict the class labels and the bounding boxes of particular objects within the images (Liu et al., 2016). Whilst FPN is an algorithm that is used to optimize the integration of information from convolutional multiple layers which helps the model to detect the object accurately (Lin et al., 2016). The combination of the MobileNetV1 with SSD and FPN is to produce a better and efficient pretrained model. This pretrained model is publicly distributed by Tensorflow.

3.2.2.2. MobileNetV1 Architecture

As it has been discussed earlier in Chapter 2, in CNN, the input data is extracted by the CNN architecture consisting of multiple layers of filters and a set of kernels to obtain its features along the learning process called convolutional layers. The same concept is applied within the MobileNetV1 algorithm in which the convolutional layers are applied to its architecture. The MobileNetV1 uses depthwise separable and pointwise convolution as convolution operations. In contrast to standard CNN layers, MobileNetV1 uses a single filter instead of multiple layers by using the depth wise separable and pointwise convolution.

The above diagram depicts the filters of standard convolutional and MobileNetV1 according to its original documentation (Howard et al., 2017). Figure 3.5 shows the regular convolutional filters that apply in the standard convolutional neural network structure. Figure 3.6 shows depthwise filters using 3 x 3 filters that later will be summed with 1 x 1 pointwise filters as it is shown in Figure 3.7.

Figure 3.5. Regular Convolution Filters.

Figure 3.5. Regular Convolution Filters.

Figure 3.6. Depthwise Convolution Filters.

Figure 3.6. Depthwise Convolution Filters.

Figure 3.7. Pointwise Convolution Filters.

Figure 3.7. Pointwise Convolution Filters.

The computational cost of the standard convolution can be expressed as follow:

$$D_K\cdot D_K\cdot M\cdot N\cdot D_F\cdot D_F$$

(1)

In this case, M is the number of the input also called as input depth, $D_K$ is the spatial dimension, $D_F$ is the spatial height and width and the N is the output number also called output depth. Meanwhile, the computational cost of the summed of the depthwise and pointwise convolution can be expressed as shown below, the summed of these two layers called as separable convolutions (Radford et al., 2018).

$$D_K\cdot D_K\cdot M\cdot D_F\cdot D_F+N\cdot M\cdot D_F\cdot D_F$$

(2)

As it is highlighted that MobileNetV1 has less computational cost by expressing two convolutional filters, the reduction of the computational can be expressed by comparing the two expressions of standard convolution and the separable convolution given as below:

$$\frac{D_K\cdot D_K\cdot M\cdot D_F\cdot D_F+N\cdot M\cdot D_F\cdot D_{Fb}}{D_K\cdot D_K\cdot M\cdot N\cdot D_F\cdot D_F}=\frac{1}{N}+\frac{1}{{D^2}_K}$$

(3)

The given expression interprets that the computational cost of the separable convolution is less compared to the standard convolution filters. In addition, according to the original publication of MobileNetV1, every layer of the convolution is followed by Batch Normalization (BN) and Rectified Linear Unit (ReLU). BN is a deep learning technique that is used to accelerate network training by normalizing the activation layers within the range of the input data batch (Ioffe & Szegedy, 2015). ReLU is a neural network activation function that allows the model to train on complex patterns as well as improving the training duration. Both BN and ReLU are used to stabilize and optimize the training of the model. The below figure shows the distinction between the structure of the standard convolution layers (left) and the separable convolution layers (right) followed by BN and ReLU after every layer according to Khasoggi et al. (2019).

Figure 3.8. Training Structure with BN and ReLU.

Figure 3.8. Training Structure with BN and ReLU.

3.2.2.3. Tensorflow Object Detection API

To use the specified pretrained model, the project requires a Tensorflow object detection model repository and object detection model API. The repository of the object detection is required as the set of dependencies that are used to run and perform the task of the model. The use of object detection API is to help the Tensorflow model to run properly as in the training and the deployment. On the other hand, it gives more feasibility for the users to run the model. The API also includes a set of tools and utilities for working with the models, such as data preparation and evaluation tools, and visualization functions (Hongkun Yu et al., 2020).

3.2.2.4. Model Deployment Using Streamlit

To use the specified pretrained model, the project requires a Tensorflow object detection model repository and object detection model API. The repository of object detection is required as the set of dependencies that are used to run and perform the task of the model. The use of object detection API is to help the Tensorflow model to run properly as in the training and the deployment. On the other hand, it gives more feasibility for the users to run the model. The API also includes a set of tools and utilities for working with the models, such as data preparation and evaluation tools, and visualization functions (Hongkun Yu et al., 2020).

3.3. Testing and Evaluation Metrics

According to the objectives and the scope of the study from Chapter 1, the evaluation of the model is determined by its ability to interpret the Malaysian sign language that refers to Bahasa Isyarat Malaysia. The measure of the performance in this project centers on the success of the recognition rate of the model. The success rate of the model or so called as score refers to the detection score that is generated within the pretrained model. The score indicates the ability of the model to identify and locate the object which in this case is the sign language. It is calculated using the Intersection over Union (IoU) score. The IoU score measures the overlap between two sets of bounding boxes, and it is calculated as the ratio of the intersection of the two bounding boxes to the union of the two bounding boxes. To conclude, the success rate comes from the functions that are generated by the pretrained model such as detection_scores function. The model was trained on data with specific characteristics of angle, subject, and background complexity, hence the testing will include various scenarios with different characteristics from the datasets to evaluate the model’s ability and limitations in interpreting the sign language.

In this test, the characteristics of the parameters test are divided into; the subject’s background, the environment background, and with the respective angle of the subject and object orientation towards the device’s camera. The subject’s background refers to the complexity of the pattern of the clothes that are worn by the subject, whilst the background refers to the environment background of the test.

The inclusion of various types of backgrounds aims to observe the effect of background of the subject and the environment on the model’s performance in recognizing the object, however this study is not discussing the metrics that affect the performance in detail. The test can be described into several scenarios as follow:

Table 3.1. Testing Scenarios.

Table 3.1. Testing Scenarios.

Test	Environment Background	Subject Background	Angle 1	Angle 2	Angle 3
Test 1	Similar	Similar	Lean Left	Straight	Lean Right
Test 2		Different
Test 3	Different	Similar
Test 4		Different

Under Test 1 and Test 2, the model recognizes the object over a similar environment background to the dataset environment background, whereas the subject’s background in Test 2 is differentiated. Under Test 3 and Test 4, the model performs the recognition of the object using a different environment background from the dataset environment background, whereas under Test 4, the subject’s background is also differentiated. Each test covers all classes of the dataset, which are “hello,” “yes,” “no,” “thanks,” and “sorry.” The test results are based on the recognition score of every attempt of each class. Each class has a maximum of five test attempts using the real time sign language recognition model in the web application using a desktop device.

Results and Discussion

4.2. Results

As it has been mentioned in Chapter 3, this project performed four tests with at least 60 attempts being done in testing the model. The results of the test are shown in Table 4.1, Table 4.2, Table 4.3, and Table 4.4. The number of the scores are based on the score that is calculated by the model in measuring the successful rate of its model in recognizing the given sign language which is based on the detection scores. Each table represents the results of Test 1, Test 2, Test 3, and Test 4 sequentially with the respective angle of the subject and object orientation towards the device’s camera.

Table 4.1. Test 1 Results.

Table 4.1. Test 1 Results.

Class	Orientation			Average Score
	Left	Straight	Right
hello	87.00	95.20	49.60	77.27
no	83.40	88.60	61.00	77.67
yes	72.20	93.80	54.00	73.33
sorry	45.80	92.40	71.20	69.80
thanks	52.40	96.00	73.60	74.00
Total Average Score				74.41

Table 4.2. Test 2 Results.

Table 4.2. Test 2 Results.

Class	Orientation			Average Score
	Left	Straight	Right
hello	66.40	83.00	67.20	72.20
no	0.00	73.00	13.00	28.67
yes	59.80	91.80	59.80	70.47
sorry	15.00	16.60	12.20	14.60
thanks	24.60	96.00	84.40	68.33
Total Average Score				50.85

Table 4.3. Test 3 Results.

Table 4.3. Test 3 Results.

Class	Orientation			Average Score
	Left	Straight	Right
hello	52.60	60.40	12.20	41.73
no	64.20	87.60	12.00	54.60
yes	0.00	92.00	18.40	36.80
sorry	64.80	86.80	44.80	65.47
thanks	0.00	93.00	27.40	40.13
Total Average Score				47.75

Table 4.4. Test 4 Results.

Table 4.4. Test 4 Results.

Class	Orientation			Average Score
	Left	Straight	Right
hello	78.40	76.00	0.00	51.47
no	26.60	50.60	0.00	25.73
yes	0.00	80.20	14.80	31.67
sorry	27.20	86.60	0.00	37.93
thanks	90.20	96.00	82.80	89.67
Total Average Score				47.29

According to the previous chapter, Test 1 in Table 4.1 tested the model on five classes where the environment and the subject backgrounds are similar to the background from the dataset. The test also involves the various orientations of the object towards the camera. Table 4.2 shows the results of Test 2, which similarly uses a similar environment background from the dataset where the subject background is different. Test 2 also includes the various orientations of the object.

Table 4.3 and Table 4.4 show the results of Test 3 and Test 4 sequentially. It shows the results of the test where the model is tested using different environment background in various orientation. As discussed earlier, the scores are taken from the average value of the detection scores of the models from five attempts of each class.

From multiple tests conducted and the results were recorded in tables. The results show that Test 1 had the highest average score of 74 percent successful rate among all the tests. Test 2, Test 3, and Test 4 had a lower average score, but the exact percentages were not specified. It is also mentioned that the class “thanks” with a straight orientation produced the highest score across all the tests. This suggests that the model or algorithm being tested performed particularly well when recognizing the “thanks” class, and that the orientation of the object had a positive impact on the performance. Across the results, the highest score achieved is 96 percent. It indicates that under certain conditions the model can accurately predict and recognize the sign language.

Conclusion

References

1. Alnaim, N. (2020). Hand gesture recognition using deep learning neural networks, Brunel University London.
2. Airehrour, D.; Gutierrez, J.; Ray, S.K. (2015, November). GradeTrust: A secure trust based routing protocol for MANETs. In 2015 International Telecommunication Networks and Applications Conference (ITNAC) (pp. 65-70). IEEE.
3. Anwesa Chaudhuri, A.C.; Sanjib Ray, S.R. (2015). Antiproliferative activity of phytochemicals present in aerial parts aqueous extract of Ampelocissus latifolia (Roxb.) Planch. on apical meristem cells.
4. Balakrishnan, S.; Ruskhan, B.; Zhen, L.W.; Huang, T.S.; Soong, W.T. Y.; Shah, I.A. Down2Park: Finding New Ways to Park. Journal of Survey in Fisheries Sciences 2023, 322–338. [Google Scholar]
5. Fujiyoshi, H.; Hirakawa, T.; Yamashita, T. Deep learning-based image recognition for autonomous driving. In IATSS Research 2019, 43. [Google Scholar] [CrossRef]
6. Goldsborough, P. (2016). A Tour of TensorFlow Proseminar Data Mining. Arxiv.
7. Gouda, W.; Almurafeh, M.; Humayun, M.; Jhanjhi, N.Z. (2022, February). Detection of COVID-19 based on chest X-rays using deep learning. In Healthcare (Vol. 10, No. 2, p. 343). MDPI.
8. Hussain, S.J.; Irfan, M.; Jhanjhi, N.Z.; Hussain, K.; Humayun, M. Performance Enhancement in Wireless Body Area Networks with Secure Communication. Wireless Personal Communications 2021, 116. [Google Scholar] [CrossRef]
9. Humayun, M.; Ashfaq, F.; Jhanjhi, N.Z.; Alsadun, M.K. Traffic management: Multi-scale vehicle detection in varying weather conditions using yolov4 and spatial pyramid pooling network. Electronics 2022, 11, 2748. [Google Scholar] [CrossRef]
10. Ioffe, S.; Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. 32nd International Conference on Machine Learning, ICML 2015, 1.
11. John, V.; Boyali, A.; Mita, S.; Imanishi, M.; Sanma, N. Deep Learning-Based Fast Hand Gesture Recognition Using Representative Frames. 2016 International Conference on Digital Image Computing: Techniques and Applications, DICTA 2016, 2016. [Google Scholar] [CrossRef]
12. Jhanjhi, N.Z.; Brohi, S.N.; Malik, N.A.; Humayun, M. (2020, October). Proposing a hybrid rpl protocol for rank and wormhole attack mitigation using machine learning. In 2020 2nd International Conference on Computer and Information Sciences (ICCIS) (pp. 1-6). IEEE.
13. Kellogg, B.; Talla, V.; Gollakota, S. (2014). Bringing gesture recognition to all devices. Proceedings of the 11th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2014.
14. Leksut, J.T.; Zhao, J.; Itti, L. Learning visual variation for object recognition. Image and Vision Computing 2020, 98. [Google Scholar] [CrossRef]
15. Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.Y.; Berg, A.C. SSD: Single shot multibox detector. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2016; 9905 LNCS. [Google Scholar] [CrossRef]
16. Lim, M.; Abdullah, A.; Jhanjhi, N.Z.; Khan, M.K.; Supramaniam, M. Link prediction in time-evolving criminal network with deep reinforcement learning technique. IEEE Access 2019, 7, 184797–184807. [Google Scholar] [CrossRef]
17. Mallick, C.; Bhoi, S.K.; Singh, T.; Swain, P.; Ruskhan, B.; Hussain, K.; Sahoo, K.S. Transportation Problem Solver for Drug Delivery in Pharmaceutical Companies using Steppingstone Method. International Journal of Intelligent Systems and Applications in Engineering 2023, 11. [Google Scholar]
18. Mitra, S.; Acharya, T. Gesture recognition: A survey. IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews, 2007; 37. [Google Scholar] [CrossRef]
19. Mujahid, A.; Awan, M.J.; Yasin, A.; Mohammed, M.A.; Damaševičius, R.; Maskeliūnas, R.; Abdulkareem, K.H. Real-time hand gesture recognition based on deep learning YOLOv3 model. Applied Sciences (Switzerland) 2021, 11. [Google Scholar] [CrossRef]
20. Neethu, P.S.; Suguna, R.; Sathish, D. An efficient method for human hand gesture detection and recognition using deep learning convolutional neural networks. Soft Computing 2020, 24. [Google Scholar] [CrossRef]
21. Oudah, M.; Al-Naji, A.; Chahl, J. Computer Vision for Elderly Care Based on Deep Learning CNN and SVM. IOP Conference Series: Materials Science and Engineering, 2021; 1105. [Google Scholar] [CrossRef]
22. Riskhan, B.; Muhammad, R. Docker in Online Education: Have the Near-native Performance of CPU, Memory and Network? International Journal of Computer Theory and Engineering 2017, 9, 290–293. [Google Scholar] [CrossRef]
23. Riskhan, B.; Safuan, H.A. J.; Hussain, K.; Elnour, A.A. H.; Abdelmaboud, A.; Khan, F.; Kundi, M. An Adaptive Distributed Denial of Service Attack Prevention Technique in a Distributed Environment. Sensors 2023, 23. [Google Scholar] [CrossRef] [PubMed]
24. Ray, S.K.; Sinha, R.; Ray, S.K. (2015, June). A smartphone-based post-disaster management mechanism using WiFi tethering. In 2015 IEEE 10th conference on industrial electronics and applications (ICIEA) (pp. 966-971). IEEE.
25. Saha, S. A Comprehensive Guide to Convolutional Neural Networks — the ELI5 way. Towards Data Science 2015.
26. Singhal, V.; Jain, S.S.; Anand, D.; Singh, A.; Verma, S.; Rodrigues, J.J. . & Iwendi, C. Artificial intelligence enabled road vehicle-train collision risk assessment framework for unmanned railway level crossings. IEEE Access 2020, 8, 113790–113806. [Google Scholar]
27. Taj, I.; Zaman, N. Towards industrial revolution 5.0 and explainable artificial intelligence: Challenges and opportunities. International Journal of Computing and Digital Systems 2022, 12, 295–320. [Google Scholar] [CrossRef]
28. Yamashita, R.; Nishio, M.; Do, R.K. G.; Togashi, K. (2018). Convolutional neural networks: an overview and application in radiology. In Insights into Imaging (Vol. 9, Issue 4). [CrossRef]
29. Zeng, Z.; Gong, Q.; Zhang, J. (2019). CNN model design of gesture recognition based on tensorflow framework. Proceedings of 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference, ITNEC 2019. [CrossRef]