1. Introduction

2. Related Work

2.3. Variational Autoencoder (VAE)[7]

Variations, or variations. What should we know about functionals before we talk about variations? To review the functions we have learned since the beginning, it is to take a given input value x, through a series of changes f(x), to get the output value y. Notice here we’re putting a number in, and we’re putting a number out. Is there a case where our argument is a function instead of a number? The classic question is, given two fixed points A and B, we can take any path from point A to point B, and find out in what path the time from point A to point B is shortest? By this point most people have the answer - the shortest line between two points. A function where the input variable is a function and the output variable is a numerical value is called a functional. The popular understanding of a functional is a function of a function.

Usually, we feed the input image into the NN Encoder and get a latent code, usually the dimension of this latent code is much smaller than the dimension of the input object, which is a compact representation of the input object. Next, we feed this latent code into [8]NN Decoder for decoding and output the reconstructed original object.

Figure 1. Auto-Encoder architecture diagram.

Figure 1. Auto-Encoder architecture diagram.

Auto-Encoder was proposed by Rumelhart in 1986 and can be used for processing high-dimensional complex data, which promotes the development of neural networks. A self-coding neural network is an unsupervised learning algorithm (training examples are not labeled) that uses the BP backpropagation algorithm and strives to make the output as close to the input as possible.

AE networks generally have two characteristics:

1.dim(Hidden layer) << dim(Input layer) : the hidden layer dimension should be much smaller than the input dimension.

2. The Output of the decoding layer is used for Reconstruction Input, so we should minimizer(Reconstruction error(Input, Output)), that is, minimize the reconstruction error between input and output.

AE algorithm description:

1.Encoder is responsible for compressing the input data, and compressing the N-dimensional input data into M-dimensional data (m << n) through the Hidden layer. In other words, encoder learns a set of parameters to obtain a latent code;

2.Decoder is responsible for restoring data and restoring original data as much as possible with the least loss when needed.

AE can be applied to data dimensionality reduction, feature extraction and data visualization analysis in machine learning, and can also be extended and applied to generative models.

Auto-Encoder for a particular generation model, it should generally satisfy the following two points:

1. Encoders and decoders can be separated independently[9] (similar to GAN Generator and Discriminator).

2. Any code sampled in a fixed dimension should be able to produce a clear and true picture through the decoder.

VAE is to add appropriate noise to the code on the original AE structure. First we input the input into the NN Encoder and calculate two sets of encodings: one is encoded as the mean encoding $\mathrm{m}=(\mathrm{m}1,\mathrm{m}2,\mathrm{m}3)$, and the other is encoded as the variance encoding $\mathsf{\sigma }=(\mathsf{\sigma }1,\mathsf{\sigma }2,\mathsf{\sigma }3).$that controls the noise interference degree. The variance code $\mathsf{\sigma }$ is mainly used to assign weight to noise code. In the figure, a layer of exponential operation is applied to variance code $\mathrm{z}=(\mathrm{e}1,\mathrm{w}2,\mathrm{e}3).$before the weight is assigned to $\mathrm{z}$, as long as the reason is that the weight value learned by NN is positive and negative, so this is to ensure that the weight assigned is positive. Finally, we overlay the original code $\mathrm{m}$ with the weighted noise code to obtain a new latent code, which is then fed into the NN Decoder. Observing the figure above, it can be seen that in addition to the reconstruction error of traditional AE, the loss function has the following additional item:

$$\mathrm{c}=(\mathrm{c}1,\mathrm{c}2,\mathrm{c}3)={\sum }_{\mathrm{i}=1}^3({\mathrm{e}}^{{\mathsf{\sigma }}_{\mathrm{i}}}-(1+{\mathsf{\sigma }}_{\mathrm{i}})+({\mathrm{m}}_{\mathrm{i}}{)}^2)$$

(1)

The principle of VAE is an unsupervised generative model, which is based on Gaussian mixture model.

Figure 2. Flow chart of VAE algorithm.

Figure 2. Flow chart of VAE algorithm.

From the model structure of VAE[10], we can see that the noise code z is a vector generated by a standard normal distribution, which we randomly sample m points, where m follows a polynomial distribution p(x). Every time we sample a point m, we map it to a Gaussian distribution$\mathrm{N}({\mathsf{\mu }}^{\mathrm{m}},{\mathsf{\sigma }}^{\mathrm{m}}$), so a polynomial distribution using the mixture model can be represented as:

$$\mathrm{p}(\mathrm{x})={\sum }_{\mathrm{m}}\mathrm{p}(\mathrm{m})\mathrm{p}(\mathrm{x}｜\mathrm{m})=\int \mathrm{z}\mathrm{p}(\mathrm{z})\mathrm{p}(\mathrm{x}｜\mathrm{z}){\mathrm{d}}_{\mathrm{z}}$$

(2)

The m~p(m),x | m~N(${\mathsf{\mu }}^{\mathrm{m}},{\mathsf{\sigma }}^{\mathrm{m}}$). After the above operation, we can convert the original discrete encoding with a large number of distorted areas into a continuous and effective encoding.

Therefore, variational autoencoders (VAE) -based anomaly detection methods identify anomalies by learning potential representations of the data. VAE is a generative model that compresses input data into a potential space via an encoder and then reconstructs the data via a decoder. The normal data is reconstructed efficiently by VAE, while the abnormal data has a large reconstruction error due to the different distribution from the training data. For example, in network traffic monitoring, VAE can detect abnormal traffic by analyzing normal traffic patterns, and when a significant reconstruction error is detected, it can be flagged as a potential network attack or failure.

The advantage of VAE lies in its high productivity and flexibility[11,12]. It is not only able to handle high-dimensional data, but also to capture complex patterns in the data, which makes it excellent in anomaly detection. VAE is more adaptable to new and unseen anomalies than traditional statistical or rule-based methods. For example, in the predictive maintenance of industrial equipment, VAE can be used to model the normal operation data of the equipment, and when the equipment exhibits abnormal behavior, VAE can quickly identify and issue alerts, thereby avoiding potential failures and downtime.

3. Methodology

3.1. Data Set

The network traffic dataset utilized in this study encompasses diverse types of IoT devices and network attack traffic. Prior to analysis, the dataset underwent preprocessing steps including data cleaning and normalization, ensuring data quality and enhancing the efficacy of model training.

Table 1. Dataset Details.

Table 1. Dataset Details.

Dataset Path	/root/yolov8-main/dataset/data2/images
Training Set	train
Validation Set	val
Test Set	test
Number of Classes	10
Class Names
0: pedestrian	Pedestrian
1: person	Person
2: car	Car
3: van	Van
4: bus	Bus
5: truck	Truck
6: motor	Motor
7: bicycle	Bicycle
8: awning-tricycle	Awning-tricycle
9: tricycle	Tricycle

The dataset, as described previously, comprises images categorized into 10 classes, representing various objects such as pedestrians, cars, vans, buses, and others, contributing to a comprehensive understanding of real-world scenarios in the domain of object detection and classification.

3.2. CNN Model

Convolutional layer is one of the core components in convolutional neural network (CNN), which is mainly responsible for extracting the features of input data (usually images). The convolution layer performs convolution operations by sliding small filters (also known as convolution cores or filters) over the input data, thereby extracting local features. These local features usually include visual information such as edges, textures, and shapes, which are crucial for subsequent tasks such as image classification and object detection.

The operation process of the convolution layer is shown below, using a convolution kernel to scan the entire image.

Figure 3. CNN model architecture.

Figure 3. CNN model architecture.

In this paper, the use of CNN model mainly lies in the effective classification and identification of network traffic. Through the four convolutional layers and pooling layers in the model, the network can learn the features in the data and map them to a higher-dimensional representation space, so as to achieve feature extraction and dimensionality reduction of network traffic. This process of feature extraction and dimensionality reduction helps to reduce the complexity of data and improve the training efficiency and classification accuracy of the model.

By training with the cross-entropy loss function and the Adam optimizer[13], the model can effectively learn the patterns and laws in the network traffic data and make accurate classification. The cross-entropy loss function is widely used in multi-class classification problems, which can measure the difference between the model prediction results and the actual labels, so as to guide the model optimization parameters to improve the classification accuracy. Adam optimizer is an adaptive learning rate optimization algorithm, which can automatically adjust the learning rate according to the gradient of parameters, accelerate the convergence speed of the model, and avoid falling into the local optimal solution.

Therefore, in this experiment, network traffic can be classified and identified efficiently through the application of CNN model, which provides strong support and guarantee for threat detection and defense in the field of network security.

3.3. Variational Autoencoder (AVE) Model

In this experiment, VAE (Variational Autoencoder) model plays an important role and has significant advantages. [14]The VAE model is composed of encoders and decoders, and its main purpose is to learn the potential representation of the data and realize the transformation of the potential space through the re-parameterization technique. In this experiment, the functions and advantages of VAE model can be further elaborated as follows:

Figure 4. Variational Autoencoder (AVE) Model.

Figure 4. Variational Autoencoder (AVE) Model.

1. Data potential representation learning: The VAE model maps the input data to the distribution in the potential space through the encoder part, and reconstructs the representation in the potential space into the original data through the decoder part. This learning process enables the model to capture the underlying structure and features in the data, helping to improve the understanding and analysis of the data.

2. Data reconstruction and generation: Through the decoder part, VAE model can generate new samples similar to the input data, so as to achieve data reconstruction and generation. This ability can be used to enhance, expand or generate new data samples in the experiment, which helps to improve the generalization ability and performance of the model.

3. Continuity and interpolation of potential space: Because VAE model learns and transforms potential space, there is continuity and interpolability between different representations in potential space. This means that in the potential space, similar data samples correspond to similar potential representations, so that data samples can be interpolated and generated in the potential space, which is conducive to the exploratory analysis and visualization of the model.

4. Optimization of loss function[15]: In this experiment, the VAE model is optimized using mean square error (MSE) and KL divergence as loss functions. The selection of the loss function helps the model to learn the reconstruction error of the data and the structure information of the potential space, thus improving the training stability and performance of the model.

To sum up, VAE model plays an important role in this experiment, such as learning data potential representation, data reconstruction and generation, exploratory analysis of potential space, and has significant advantages in loss function optimization, data enhancement and model generalization.

4. Experiment

4.3. Experimental Procedure

The training process for both the Convolutional Neural Network (CNN) and the Variational Autoencoder (VAE) involves several key steps. Firstly, the data is loaded and preprocessed, with features extracted and normalized as necessary. Subsequently, forward propagation occurs through the respective models, where input data passes through layers of neurons, applying activation functions and transformations. During backward propagation, the loss is computed, typically using functions like CrossEntropyLoss for CNN and a combination of reconstruction loss and KL divergence for VAE. Gradients of the loss with respect to model parameters are then calculated, allowing the optimizer (such as Adam) to update the model parameters iteratively. This process repeats for each batch of data for a specified number of epochs, gradually optimizing the model parameters towards better performance.

Following the training phase, the CNN model’s performance is evaluated on the test dataset. The model’s accuracy, visualized in “Accuracy.png,” illustrates its ability to correctly classify IoT device data. Specifically, the CNN model achieved an impressive accuracy of 95.85% on the test set. This high accuracy underscores the effectiveness of the CNN architecture in accurately classifying different types of IoT device data, showcasing its potential utility in real-world applications for anomaly detection and classification tasks.

4.4. Experimental Result

To succinctly represent the detection results for different types of traffic, we will select three images that effectively capture the detection outcomes across diverse IoT device categories:

The detection results for various types of IoT device traffic are represented through three key images. Firstly, “gafgyt_danmini_doorbell(a)” illustrates the detection outcomes for traffic associated with the Gafgyt malware targeting Danmini Doorbell devices, showcasing anomalies in this commonly targeted IoT device category. Secondly, “gafgyt_ecobee_thermostat(b)” displays the detection results for traffic related to the Gafgyt malware affecting Ecobee Thermostat devices, shedding light on anomalies in another significant IoT device category often exploited by malware. Lastly, “mirai_provision_security_camera(c)” demonstrates the detection outcomes for traffic associated with the Mirai malware targeting Provision Security Camera devices, providing insights into the detection results for a distinct type of IoT device affected by a different malware variant. Together, these images offer a comprehensive overview of the detection performance across diverse IoT device categories and malware types, highlighting the effectiveness of the anomaly detection system in identifying and flagging suspicious traffic patterns.

5. Conclusion

References

1. Zhan, X., Shi, C., Li, L., Xu, K., & Zheng, H. (2024). Aspect category sentiment analysis based on multiple attention mechanisms and pre-trained models. Applied and Computational Engineering, 71, 21-26. [CrossRef]
2. Wu, B., Xu, J., Zhang, Y., Liu, B., Gong, Y., & Huang, J. (2024). Integration of computer networks and artificial neural networks for an AI-based network operator. Applied and Computational Engineering, 64, 115-120. [CrossRef]
3. Liang, P., Song, B., Zhan, X., Chen, Z., & Yuan, J. (2024). Automating the training and deployment of models in MLOps by integrating systems with machine learning. Applied and Computational Engineering, 67, 1-7.
4. Li, A., Yang, T., Zhan, X., Shi, Y., & Li, H. (2024). Utilizing Data Science and AI for Customer Churn Prediction in Marketing. Journal of Theory and Practice of Engineering Science, 4(05), 72-79.
5. Wu, B., Gong, Y., Zheng, H., Zhang, Y., Huang, J., & Xu, J. (2024). Enterprise cloud resource optimization and management based on cloud operations. Applied and Computational Engineering, 67, 8-14. [CrossRef]
6. Xu, J., Wu, B., Huang, J., Gong, Y., Zhang, Y., & Liu, B. (2024). Practical applications of advanced cloud services and generative AI systems in medical image analysis. Applied and Computational Engineering, 64, 82-87.
7. Huang, J., Zhang, Y., Xu, J., Wu, B., Liu, B., & Gong, Y. Implementation of Seamless Assistance with Google Assistant Leveraging Cloud Computing.
8. Zhang, Y., Liu, B., Gong, Y., Huang, J., Xu, J., & Wan, W. (2024). Application of machine learning optimization in cloud computing resource scheduling and management. Applied and Computational Engineering, 64, 9-14.
9. Shi, Y., Li, L., Li, H., Li, A., & Lin, Y. (2024). Aspect-Level Sentiment Analysis of Customer Reviews Based on Neural Multi-task Learning. Journal of Theory and Practice of Engineering Science, 4(04), 1-8.
10. Shi, Y., Yuan, J., Yang, P., Wang, Y., & Chen, Z. Implementing Intelligent Predictive Models for Patient Disease Risk in Cloud Data Warehousing. [CrossRef]
11. Zhan, T., Shi, C., Shi, Y., Li, H., & Lin, Y. (2024). Optimization Techniques for Sentiment Analysis Based on LLM (GPT-3). arXiv preprint arXiv:2405.09770.
12. Li, Huixiang, et al. “AI Face Recognition and Processing Technology Based on GPU Computing.” Journal of Theory and Practice of Engineering Science 4.05 (2024): 9-16.
13. Yuan, J., Lin, Y., Shi, Y., Yang, T., & Li, A. (2024). Applications of Artificial Intelligence Generative Adversarial Techniques in the Financial Sector. Academic Journal of Sociology and Management, 2(3), 59-66.
14. Lin, Y., Li, A., Li, H., Shi, Y., & Zhan, X. (2024). GPU-Optimized Image Processing and Generation Based on Deep Learning and Computer Vision. Journal of Artificial Intelligence General science (JAIGS) ISSN: 3006-4023, 5(1), 39-49.
15. Chen, Zhou, et al. “Application of Cloud-Driven Intelligent Medical Imaging Analysis in Disease Detection.” Journal of Theory and Practice of Engineering Science 4.05 (2024): 64-71.
16. Wang, B., Lei, H., Shui, Z., Chen, Z., & Yang, P. (2024). Current State of Autonomous Driving Applications Based on Distributed Perception and Decision-Making.
17. Jiang, W., Qian, K., Fan, C., Ding, W., & Li, Z. (2024). Applications of generative AI-based financial robot advisors as investment consultants. Applied and Computational Engineering, 67, 28-33.
18. Ding, W., Zhou, H., Tan, H., Li, Z., & Fan, C. (2024). Automated Compatibility Testing Method for Distributed Software Systems in Cloud Computing.
19. Fan, C., Li, Z., Ding, W., Zhou, H., & Qian, K. Integrating Artificial Intelligence with SLAM Technology for Robotic Navigation and Localization in Unknown Environments. [CrossRef]
20. Guo, L., Li, Z., Qian, K., Ding, W., & Chen, Z. (2024). Bank Credit Risk Early Warning Model Based on Machine Learning Decision Trees. Journal of Economic Theory and Business Management, 1(3), 24-30.
21. Li, Zihan, et al. “Robot Navigation and Map Construction Based on SLAM Technology.” (2024).
22. Fan, C., Ding, W., Qian, K., Tan, H., & Li, Z. (2024). Cueing Flight Object Trajectory and Safety Prediction Based on SLAM Technology. Journal of Theory and Practice of Engineering Science, 4(05), 1-8.
23. Srivastava, S., Huang, C., Fan, W., & Yao, Z. (2023). Instance Needs More Care: Rewriting Prompts for Instances Yields Better Zero-Shot Performance. arXiv preprint arXiv:2310.02107.
24. Xiao, J., Wang, J., Bao, W., Deng, T. and Bi, S., Application progress of natural language processing technology in financial research. [CrossRef]
25. .
26. Ding, W., Tan, H., Zhou, H., Li, Z., & Fan, C. Immediate Traffic Flow Monitoring and Management Based on Multimodal Data in Cloud Computing.
27. Qian, K., Fan, C., Li, Z., Zhou, H., & Ding, W. (2024). Implementation of Artificial Intelligence in Investment Decision-making in the Chinese A-share Market. Journal of Economic Theory and Business Management, 1(2), 36-42.
28. Tian, J., Li, H., Qi, Y., Wang, X., & Feng, Y. (2024). Intelligent Medical Detection and Diagnosis Assisted by Deep Learning. Applied and Computational Engineering, 64, 121-126.
29. Zhou, Y., Zhan, T., Wu, Y., Song, B., & Shi, C. (2024). RNA Secondary Structure Prediction Using Transformer-Based Deep Learning Models. arXiv preprint arXiv:2405.06655. [CrossRef]
30. Liu, B., Cai, G., Ling, Z., Qian, J., & Zhang, Q. (2024). Precise Positioning and Prediction System for Autonomous Driving Based on Generative Artificial Intelligence. Applied and Computational Engineering, 64, 42-49. [CrossRef]
31. Cui, Z., Lin, L., Zong, Y., Chen, Y., & Wang, S. (2024). Precision Gene Editing Using Deep Learning: A Case Study of the CRISPR-Cas9 Editor. Applied and Computational Engineering, 64, 134-141.
32. Yu, D., Xie, Y., An, W., Li, Z., & Yao, Y. (2023, December). Joint Coordinate Regression and Association For Multi-Person Pose Estimation, A Pure Neural Network Approach. In Proceedings of the 5th ACM International Conference on Multimedia in Asia (pp. 1-8).
33. Wang, B., He, Y., Shui, Z., Xin, Q., & Lei, H. (2024). Predictive Optimization of DDoS Attack Mitigation in Distributed Systems using Machine Learning. Applied and Computational Engineering, 64, 95-100. [CrossRef]
34. He, Zheng, et al. “Application of K-means clustering based on artificial intelligence in gene statistics of biological information engineering.”.