1. Introduction

2. Abnormal Behaviors of Pedestrians in Public Places

3. Challenges Brought by Crowd Density

4. Research Status of Crowd Abnormal Behavior Recognition

4.2. Deep Learning-Based Methods

In recent years, deep learning has become the core driving force in the field of computer vision. Compared with traditional methods, deep learning mostly uses the powerful representational learning ability of deep neural networks to automatically extract high-level abstract features from raw data, which can effectively distinguish normal behaviors from abnormal behaviors. According to the network structure and flow processing method, deep learning abnormal behavior recognition methods are classified into the following categories: abnormal behavior recognition methods based on convolutional neural networks, based on autoencoders, based on generative adversarial networks, based on long short-term memory networks and based on self-attention mechanisms.

4.2.1. Methods Based on Convolutional Neural Network

The Convolutional Neural Network (CNN), due to its strong ability to process pixel data and efficient learning feature representation, has become an indispensable part of the field of computer vision, such as image classification, semantic segmentation, and target feature analysis. As a deep, feed-forward neural network system, the operation process of CNN can be divided into two major steps: feature extraction and classification decision-making, and the structure is shown in Figure 4. In the field of crowd abnormal behavior recognition, the current CNN-based methods mainly focus on two core directions: spatial feature and spatiotemporal feature learning.

1) Spatial feature learning focuses on extracting the appearance information of pedestrians from video frames to identify atypical behavior patterns in appearance, that is, appearance anomalies. Based on different video forms and annotations, crowd abnormal behavior recognition methods can be divided into three learning methods: supervised, semi-supervised, and unsupervised.

Supervised abnormal behavior recognition is a strategy based on annotated datasets. It extracts pedestrian features through CNN and builds a classification model to identify abnormal patterns in the original data based on normal and abnormal behavior labels [42]. Since explicit category labels are provided during training, CNN can learn precise feature representations for specific tasks, thereby achieving a high accuracy rate for labeled abnormal behavior recognition. However, this method highly depends on the annotated datasets of abnormal behaviors, and the cumbersome manual annotation limits the development of supervised algorithms.

Unsupervised/weakly supervised learning methods can use CNN to extract and analyze pedestrian feature representations without explicit abnormal behavior category labels or with only a small number of labels. This method can detect unknown or unlabeled abnormal behavior types, adapt to a wider range of behavior patterns, and improve the generalization ability of the model in practical applications. Unsupervised/weakly supervised abnormal behavior recognition is implemented based on two stages: feature extraction and abnormal recognition. In the feature extraction stage, pedestrian appearance features in the video sequence are extracted through pre-trained CNNs, commonly including VGG [43], ResNet50 [44], AlexNet [45], GoogLeNet [46], Inception [47], etc. After extracting the appearance features of pedestrians, abnormal classification is performed through abnormal behavior recognition algorithms, commonly including a one-class classifier [48], Gaussian classifier [49], Support Vector Machine (SVM) [50], etc. Singh et al. [51] proposed a method of Aggregation of Ensembles (AOE), which fused and fine-tuned the three networks of AlexNet, GoogLeNet and VGG, especially adjusted the number of output nodes of the fully connected layer to match the binary classification task, and adopted a hierarchical fine-tuning strategy, only updating the learning rate of specific layers and keeping other layers unchanged, thereby avoiding training the network from scratch and improving the efficiency of abnormal behavior recognition. At the same time, each sub-ensemble is composed of a specific classifier and the combination of three CNN models, providing multiple classification outputs for each video frame. To solve the problem of low efficiency of video frame block convolution, Sabokrou et al. [52] proposed a method based on a Fully Convolutional Neural Network (FCN), extracting features of all regions from the entire video frame, and performing convolution and pooling operations in parallel, which can achieve a processing speed of about 370 frames per second, meeting the requirements of real-time processing.

2) Spatiotemporal feature learning methods integrate dynamic information in the time series based on space. This is usually achieved by using optical flow data or upgrading to 3D convolution technology, aiming to capture the complex relationship between motion and time between video frames and identify abnormalities in the movement trajectories and speeds of pedestrians. Under this framework, according to the different time-perception feature fusion strategies, it is mainly divided into two implementation approaches: two-stream CNN architecture and a three-dimensional convolutional neural network (3D CNN).

The two-stream convolutional neural network decomposes the processing of video data into spatial stream and temporal stream. The spatial stream focuses on the static content of the video frame, that is, the appearance feature in each frame image. The temporal stream focuses on analyzing the temporal changes and dynamic behaviors in the video sequence and usually uses optical flow images as input. Hu et al. [53] proposed a weakly supervised learning ABDL framework. Firstly, the Faster R-CNN network is used to identify objects (such as pedestrians) in the scene, then the large-scale optical flow histogram (HLSOF) is used to describe the behavior features of the objects, and finally, the behaviors are classified through the multi-instance support vector machine (MISVM) to distinguish normal or abnormal. Wang et al. [54] designed a two-stream convolutional neural network model (TS-CNN), as shown in Figure 5. Combining the above optical flow features and trajectory information, this network can process spatiotemporal information simultaneously and improve the accuracy of abnormal behavior detection. At the same time, the Kanade-Lucas-Tomasi (KLT) tracking algorithm is used to obtain the single-frame image of the crowd movement trajectory. This strategy enables the algorithm to better handle the occlusion problem in the crowd and further enhances the recognition of abnormal behaviors through the continuity feature of the behavior.

3D CNN adds a time dimension based on the original 2D convolution. By performing convolution on the time axis, 3D CNN can capture the change patterns of behaviors over time, which is extremely crucial for identifying abnormal behaviors (such as falling, running, gathering, etc.) that need to consider the behavior sequence and time context. Hu et al. [55] proposed a method based on a Deep Spatiotemporal Convolutional Neural Network (DSTCNN), extending two-dimensional convolution to three-dimensional space, and comprehensively considering the spatial features of static images and the temporal features between front and rear frames. Firstly, the video screen is divided into multiple sub-regions, and spatiotemporal data samples are extracted from these sub-regions and input into DSTCNN for training and classification, achieving accurate detection and location of abnormal behaviors. At the same time, to solve the problems of network dispersion and insufficient recognition ability, Gong et al. [56] proposed the LDA-Net framework, as shown in Figure 6. It consists of a human body detection module and an abnormal detection module. Among them, the YOLO algorithm is introduced in the human body detection module to make the abnormal detection module more concentrated; in the abnormal detection module, 3D CNN is used to simultaneously capture the motion information of labeled normal and abnormal action sequences from the spatial and temporal dimensions.

4.2.2. Methods Based on Autoencoders

Autoencoders (AE) [57] is an unsupervised learning method. The basic architecture consists of two parts, the encoder and the decoder. It reconstructs the original input by learning the effective compressed representation of the data. The structure is shown in Figure 7. In anomaly detection, AEs are trained to reconstruct normal behavior data. Thus, for those inputs significantly different from the training data (i.e., abnormal behaviors), their reconstruction errors will increase. During this process, the Mean Square Error (MSE) [58] is commonly used as the loss function. MSE is a statistical indicator that measures the difference between the predicted value and the true value, and the formula is expressed as:

Among them, ${\widehat{y}}_i$ is the predicted value of the model for the i_th sample, y_i is the true value of this sample, and ${\widehat{y}}_i-y_i$ is the residual (the difference between the predicted value and the true value). The smaller the value of MSE, the smaller the average gap between the predicted value and the true value of the model, and the higher the fitting degree of the model. In recent years, autoencoders have been widely used for crowd abnormal behavior recognition. These methods are mainly divided into two categories [59]: methods based on similarity measurement and methods based on hidden feature representation learning.

1) Methods based on similarity measurement are techniques used to evaluate and quantify the degree of similarity between two data objects, samples, sets, or vectors. In the field of crowd abnormal behavior recognition, abnormal behavior recognition is carried out by comparing the similarity between the original input image and the reconstructed image by the Convolutional Autoencoder (CAE), and the process is shown in Figure 8 The similarity between the original image and the reconstructed image is compared through similarity measurement techniques (such as Euclidean Distance [60], Pearson Correlation Coefficient [61], etc.). If the similarity is low, it indicates that there is abnormal behavior in the original image.

The original video sequence contains the appearance features of pedestrians, and the use of optical flow analysis can describe the dynamic trajectories of foreground targets. Information fusion can be carried out through the Convolutional Autoencoder (CAE) to extract the appearance attributes and dynamic features of pedestrians. Nguyen et al. [62] designed a CNN combining CAE and U-Net, which share the same encoder. CAE is responsible for learning the normal appearance structure, while U-Net attempts to associate these structures with the corresponding motion templates. This design aims to jointly improve the detection ability of abnormal frames through two complementary streams (one dealing with appearance and the other dealing with motion), and the model supports end-to-end training.

In addition to the fusion of optical flow analysis, some recent studies have utilized the processing ability of the Convolutional LSTM network (ConvLSTM) [63] for time series and combined it with the Convolutional Autoencoder (CAE), significantly improving the extraction ability of spatiotemporal features in videos. Xiao et al. [64] designed the Probabilistic Memory Autoencoder Network (PMAE), integrating 3D causal convolution and time dimension shared fully connected layers in the autoencoder network to extract spatiotemporal features in video frames, while ensuring the temporal sequence of information and avoiding the leakage of future information. Based on the spatiotemporal encoder, Nawaratne et al. [65] proposed the method of Incremental Spatiotemporal Learner (ISTL) to online learn the spatiotemporal normal behavior patterns in video surveillance streams, thereby achieving abnormal detection and location. At the same time, an active learning strategy of fuzzy aggregation is introduced to dynamically adapt to unknown or emerging normal behavior patterns in surveillance videos, allowing the system to continuously refine the understanding of normal and abnormal behaviors based on environmental changes and the acquisition of new information.

2) Methods based on hidden feature representation learning focus on using the encoder to map high-dimensional input data to a low-dimensional vector space, focusing on extracting the core information of normal behavior patterns. Subsequently, the decoder attempts to reconstruct the original data. During the optimization process, it ensures that the model captures the key structure of the behavior data. The process is shown in Figure 9. Common autoencoders can be divided into: Stacked Denoising Autoencoders (SDAE) [66], and Variational Autoencoders (VAE) [67].

SDAE extracts higher-level feature representations by combining multiple Denoising Autoencoders (DAE) [68]. The goal of a single DAE is to reconstruct the original data on noisy data. The main principle: ① The encoder maps the noisy input data to a low-dimensional hidden representation; ② The decoder reconstructs the original noise-free data based on this and minimizes the reconstruction error to learn effective information. During the stacking process, the output of the first DAE (i.e., the denoised representation) serves as the input of the second DAE, and so on. Each layer attempts to further remove noise or extract more abstract features from the output of the previous layer. Wang et al. [69] used two SDAEs to learn the appearance features and motion features of behaviors respectively. For the appearance feature, the input is the image block within the spatiotemporal volume around the dense trajectory; the motion feature is based on the optical flow block. Each SDAE contains multiple encoding layers, and the number of nodes is halved layer by layer until the bottleneck layer is reached. The output of this bottleneck layer is regarded as the learned deep feature. The structure is shown in Figure 10.

VAE is a generative model based on a probabilistic framework. Its encoder is similar to the traditional encoder, mapping the input x to a pair of vectors μ and σ, representing the mean and standard deviation of the posterior distribution q(z|x), respectively. The decoder receives the latent variable z and attempts to reconstruct the original input x, that is, estimating p(x|z). The output of the decoder can be regarded as the probability distribution of x given z. Wang et al. [70] designed a new method called S2-VAE. This method combines the stacked fully connected variational autoencoder (SF-VAE) and the skip convolutional variational autoencoder (SC-VAE). Among them, SF-VAE is a shallow generative network designed to simulate Gaussian mixture models to adapt to the actual data distribution. Filter obvious normal samples and improve the speed of abnormal detection. SC-VAE is a deep generative network that integrates low/middle/high-level features and reduces the loss in the information transmission process by fusing the features between the encoder and decoder layers through skip connections.

4.2.3. Methods Based on Generative Adversarial Networks

Generative adversarial networks (GAN) learn the data distribution through the adversarial process of the generator and the discriminator. In anomaly detection, a GAN can be trained to effectively generate only normal behavior data, and then use the reconstruction error of the generator or the output of the discriminator to determine whether the input data is normal. Abnormal behaviors are not within the distribution range of the training data and have poor generation or reconstruction effects, and thus are identified. The GAN model structure is shown in Figure 12. The abnormal behavior recognition strategies based on GAN can be roughly classified into two categories: direct reconstruction or prediction error detection method, and enhanced reconstruction method combined with autoencoder.

The core idea of the reconstruction-based method is to train a model to learn the distribution representation of normal video data. In the testing stage, it is determined whether the test sample is abnormal based on its reconstruction error. Specifically, first, a deep learning model, that is, a neural network g, is constructed. Its goal is to learn a mapping relationship so that for any given video segment or video frame x, the output g(x) processed by g is as close as possible to the original input x. Secondly, an error function f is designed, which can quantify the difference between x and g(x), that is, the reconstruction error [71] ε = f(x, g(x)). Then, during the training process, efforts are made to find an optimal set of neural network parameters so that for all samples x in the entire dataset, the corresponding sum (or average, weighted sum, etc.) of the reconstruction errors ε reaches the minimum. Song et al. [72] proposed an Ada-Net network architecture that integrates an attention-based autoencoder with a GAN model. This architecture can adaptively learn normal behavior patterns from video data and enhance the reconstruction ability of the autoencoder through adversarial learning, making the reconstructed video frames indistinguishable from the original frames. Different from the traditional reconstruction error metric based on Euclidean distance, the researchers introduced adversarial loss for the frame discriminator to make the reconstructed frame highly similar to the original frame, thereby improving the reconstruction accuracy of the autoencoder. In addition, Chen et al. [73] proposed an anomaly detection model called NM-GAN, which consists of a reconstruction network R and a discrimination network D, forming a GAN-like architecture. NM-GAN builds an end-to-end framework, mainly consisting of three modules: (1) An image-to-image encoder-decoder reconstruction network with appropriate generalization ability; (2) A CNN-based discrimination network for identifying the spatial distribution pattern of the reconstruction error map; (3) An estimation model for quantitatively scoring anomalies. The entire model is trained in an unsupervised manner.

The prediction-based method is based on the fact that a continuous sequence of normal videos has a certain contextual dependence and regularity. Abnormal behaviors are identified by comparing the difference between the observed test frame and its predicted frame. Specifically, Given the consecutive t frames of video x₁, x₂, …, x_t, the goal of the prediction model is to generate the next frame ${\widehat{x}}_{t+1}$ and strive to make ${\widehat{x}}_{t+1}$ as close as possible to the actual next frame ${\widehat{x}}_{t+1}$. In the testing stage, it is determined whether the current video frame is abnormal by comparing the difference (prediction error) between the ${\widehat{x}}_{t+1}$ predicted by the model and the actual x_t+1. Specifically, let h represent the prediction model, then ${\widehat{x}}_{t+1}=h(x_1,x_2,\mathrm{...},x_t)$. Prediction frameworks include unidirectional prediction and bidirectional prediction: Unidirectional prediction is usually based on predicting the current frame from the previous few frames. For example, Tang et al. [74] proposed the first method that combines future frame prediction and reconstruction, which is implemented through an end-to-end network. The network consists of two continuously connected U-Net modules. The first module performs future frame prediction based on the input image sequence, and the second module reconstructs the future frame based on the predicted intermediate frame, making normal and abnormal behaviors easier to distinguish in the feature space. Lee et al. [75] used the Bidirectional Multi-scale Aggregation Network (BMAN) for abnormal behavior recognition. By using bidirectional multi-scale aggregation and attention-based feature encoding, normal patterns including object scale changes and complex motions are learned. Based on the learned normal patterns, abnormal events are detected by simultaneously analyzing the appearance characteristics and motion characteristics of the scene through an appearance-motion joint detector.

The enhanced reconstruction method combining autoencoder embeds the adversarial training logic of GAN in the training framework of AE. The autoencoder combined with GAN can not only minimize the reconstruction error but also be linked with the discriminator in GAN to output a data distribution closer to the real one. The process is shown in Figure 12. A discriminator is integrated into the architecture of the autoencoder to compare and distinguish the reconstructed image of the autoencoder network and the original input image. The quality of the reconstructed image of the autoencoder network is enhanced through adversarial training. Schlegl et al. [76] proposed a method of fast anomaly generative adversarial network (f-AnoGAN), training a Wasserstein GAN through normal samples, and then training an encoder to map the image to the latent space to achieve fast inference and anomaly detection. During the anomaly detection process, the input image is reconstructed through the encoder and generator, and the combined score of the image reconstruction residual and the discriminator feature residual is used as a reliable indicator of anomaly.

4.2.4. Methods Based on Long Short-Term Memory Network

Compared with feedforward neural networks, Recurrent Neural Network (RNN) shows significant advantages in processing sequence data, and is particularly good at understanding and processing time series information. Long Short-Term Memory (LSTM) network, as an improvement and extension of RNN, is mainly designed to solve the problems of gradient vanishing and gradient explosion encountered in the training process of long sequence data. LSTM captures and analyzes the changing patterns of behaviors over time, and automatically identifies abnormal sequences that do not conform to it by learning the patterns of normal behavior sequences. LSTM introduces a gate mechanism for controlling the flow and loss of features, including three gating mechanisms: input gate, forget gate, and output gate. The structure is shown in Figure 13.

As shown in the figure above, Xt represents the input at the current moment, ht-1 and ht represent the output of the previous unit and the current unit of LSTM, Ct-1, and Ct represent the state of the previous unit and the current unit, σ represents the sigmoid activation function, and tanh represents the hyperbolic tangent activation function.

1) The forget gate determines the information to be retained. Based on ht-1 and Xt, the forget gate outputs a number between 0 and 1 for the state Ct-1. 0 means elimination and 1 means retention. The mathematical expression is:

2) The input gate determines the newly incoming information. Composed of the sigmoid function and the tanh function, the result of multiplying the output values is used to update the state. The mathematical expression is as follows:

3) The output gate determines the output information. The output state is determined through the sigmoid layer, and the output result of the state through the tanh layer is multiplied by the output result of the sigmoid layer. The mathematical expression is as follows:

In the field of crowd abnormal behavior recognition, researchers usually combine LSTM with other technologies to improve the recognition rate and processing speed of crowd abnormal behaviors. Meng et al. [77] integrated the focal loss function into the LSTM algorithm, optimized the model’s attention to abnormal samples, and reduced the model loss. This improvement not only enhanced the sensitivity of the LSTM algorithm to abnormal behaviors but also effectively improved the detection accuracy. At the same time, the author also improved the ViBe image foreground extraction method by initializing the background model and randomly selecting the neighborhood sample set to accurately extract the foreground target, thereby simplifying the scene complexity. Wu et al. [78] combined the FCN with strong spatial feature extraction ability and the LSTM with excellent time series modeling ability. The structure is shown in Figure 14. FCN is responsible for extracting high-level semantic features from video frames, while LSTM is used to capture the changing patterns of these features over time. By simultaneously learning the spatial and temporal information in the video, it effectively distinguishes normal and abnormal behaviors in the video. Sabih et al. [79] proposed an improved end-to-end supervised learning method that combines Long Short-Term Memory Network (LSTM) and Convolutional Neural Network (CNN), especially in processing optical flow features. CNN is used to extract the frame-level optical flow features calculated based on the Lucas-Kanade algorithm from the video, while the bidirectional LSTM captures the time series information of these features to jointly perform crowd abnormal detection. This method helps to understand the normal behavior patterns in the video and identify abnormal instances.

4.2.5. Methods Based on Self-Attention Mechanism

In the field of deep learning, the attention mechanism (Self-Attention SA) [80] can be interpreted as a transformation process involving the mapping between a query vector and a series of key-value pairs, thereby generating an output vector. This output vector is obtained by performing a weighted sum of each Value vector, where the weight attached to each Value vector is determined based on the quantitative evaluation of the correlation between its corresponding Key vector and the main query vector. The calculation of the attention mechanism can be divided into three stages, as shown in Figure 15.

Stage 1 is to calculate the similarity between Query and Key. Common methods include the vector dot product, Cosine similarity, and MLP network. The expressions are as follows:

Vector dot product

1) Cosine similarity

2) MLP network

Stage 2 introduces SoftMax for normalization to sort out the probability distribution of all elements. The expression is as follows:

Stage 3 performs weighted summation to obtain the Attention value. The expression is as follows:

Self-attention is a special form of the attention mechanism. Its uniqueness lies in that when calculating the attention weight, the query, key, and value all come from different parts of the same input sequence. It was first widely used in the “Transformer” model, which greatly promoted the development of the field of natural language processing, and is currently also widely used in crowd sequence modeling and abnormal behavior recognition tasks. Self-attention can not only capture the long-distance dependency within the sequence but also remain efficient in parallel computing, thereby overcoming some limitations of the Recurrent Neural Network (RNN) model.

Ye et al. [81] designed a Self-Attention Feature Aggregation (SAFA) module. The structure is shown in Figure 16. This module regenerates the feature map by aggregating the embedding representations with similar information according to the attention map. At the same time, the prior bias on normal data is minimized through the self-suppression strategy, and it is used in combination with the pixel-level frame prediction error to jointly detect abnormal frames, enhancing the ability to recognize abnormalities.

Zhang et al. [82] proposed an autoencoder model that integrates the global self-attention mechanism to capture the interaction information between the overall features of the image, facilitating the model to understand the context in which the behavior occurs and improving the accuracy of abnormal behavior determination. At the same time, a memory module is embedded in the bottleneck layer of the autoencoder to constrain the model’s over-generalization of normal behaviors.

Zhang et al. [83] fused the attention mechanism and the bidirectional Long Short-Term Memory Autoencoder Network (SABiAE). The structure is shown in Figure 17. The encoder with the self-attention mechanism captures global appearance features, and the self-attention bidirectional LSTM network is used to reduce the loss of target features, thereby extracting the information between video frames globally. The model reconstructs the frame through the decoding process and determines abnormal behaviors based on the reconstruction error.

Zhang et al. [84] introduced the self-attention mechanism in the Generative Adversarial Network (GAN), using the autoencoder containing the dense residual network and the self-attention mechanism as the generator. At the same time, a new discriminator is proposed, which integrates the self-attention module based on the relative discriminator, thereby avoiding the problem of gradient vanishing during the training process and ensuring that the frames generated by the generator are closer to the real frames.

Singh et al. [85] proposed an Attention-guided Generator and Dual Discriminator Adversarial Network (A2D-GAN), the structure is shown in Figure 19, for real-time video anomaly detection. A2D-GAN utilizes the encoder-decoder architecture, in which the encoder adds multi-level self-attention to focus on key regions and capture context information, while the decoder uses channel attention to emphasize important features and identify abnormal patterns specific to each frame.

In addition, to solve problems such as scale changes and occlusions in the crowd, fusion models such as multi-scale feature attention fusion networks and visual occlusion resolution have emerged for crowd abnormal behavior detection. For example, Sharma et al. [86] introduced a scale-aware attention module in CNN, using the cascade of multiple self-attention branches to enhance the recognition ability for scale changes. This architecture integrates motion information, motion influence maps, and features based on the energy level distribution to achieve frame-level abnormal behavior analysis. Zhao et al. [87] proposed a Dynamic Pedestrian Centroid Model (DCM). The key points of the human skeleton are extracted from the image, the pedestrian joint sub-segments are constructed accordingly, and the dynamic characteristics of pedestrians in the video sequence are described mathematically. Experiments show that DCM can detect the U-turn behavior 277ms earlier on average, and detect the fall behavior 562ms earlier on average to find abnormalities. At the same time, to solve the problem of partial occlusion of pedestrians, this paper also designed an anti-occlusion algorithm, as shown in the schematic in Figure 19. Firstly, cluster the key points of the pedestrian skeleton, divide the 21 key points into the head, upper body, and lower body, calculate the coordinates of the occluded part, estimate the effective position combined with the mechanical principle, and apply it in combination with DCM. This method has been verified by the fall behavior detection experiment, and the AUC on the 50 Volu. The U-turn dataset reaches 82.13%, and the AUC on the 50 Volu. Fall-down dataset reaches 86.70%.

In summary, Table 3 summarizes the design ideas, advantages, and disadvantages of the above various deep learning-based abnormal behavior recognition technologies. Overall, these deep learning-based methods benefit from the powerful learning ability of neural networks and can be applied to the recognition of abnormal behaviors of pedestrians in various crowd scenarios. At the same time, a strategy of using multiple methods in combination can be adopted to further improve the generalization ability and recognition accuracy of the model.

5. Comparative Analysis of Different Algorithm Experiments

6. Summary and Research Prospects

References

1. Haghani, M.; Lovreglio, R. Data-based tools can prevent crowd crushes. Science 2022, 378, 1060–1061. [Google Scholar] [CrossRef] [PubMed]
2. Luo, L.; Xie, S.; Yin, H.; Peng, C.; Ong, Y.-S. Detecting and Quantifying Crowd-level Abnormal Behaviors in Crowd Events. IEEE Trans. Inf. Forensics Secur. 2024, PP, 1–1. [Google Scholar] [CrossRef]
3. Fadhel, M.A.; Duhaim, A.M.; Saihood, A.; Sewify, A.; Al-Hamadani, M.N.; Albahri, A.; Alzubaidi, L.; Gupta, A.; Mirjalili, S.; Gu, Y. Comprehensive systematic review of information fusion methods in smart cities and urban environments. Inf. Fusion 2024, 107. [Google Scholar] [CrossRef]
4. Zhang Bingbing, Ge Shuyu, Wang Qilong, Li Peihua. Research on behavior recognition method based on multi-order information fusion. Acta Automatica Sinica, 2021, 47(3): 609-619. [CrossRef]
5. Jiang Jun, Zhang Zhuojun, Gao Mingliang, et al. A crowd abnormal behavior detection algorithm based on pulse line flow convolutional neural network. Journal of Engineering Science and Technology, 2020, 52(6):215-222.
6. Xiao Jinsheng, Shen Mengyao, Jiang Mingjun, Lei Junfeng, Bao Zhenyu. Abnormal behavior detection in surveillance videos by fusing bag attention mechanism. Acta Automatica Sinica, 2022, 48(12): 2951−2959. [CrossRef]
7. Alam, E.; Sufian, A.; Dutta, P.; Leo, M. Vision-based human fall detection systems using deep learning: A review. Comput. Biol. Med. 2022, 146, 105626. [Google Scholar] [CrossRef]
8. Yang Fan. Xiao Bin, Yu Zhiwen. Review of anomaly detection and modeling in surveillance videos. Journal of Computer Research and Development, 2021(012):058.
9. Li Y C, Jia R S, Hu Y X, et al. A Weakly-Supervised Crowd Density Estimation Method Based on Two-Stage Linear Feature Calibration. IEEE/CAA Journal of Automatica Sinica, 2024, 11(4): 965-981.
10. Zhao, R.; Dong, D.; Wang, Y.; Li, C.; Ma, Y.; Enriquez, V.F. Image-Based Crowd Stability Analysis Using Improved Multi-Column Convolutional Neural Network. IEEE Trans. Intell. Transp. Syst. 2021, 23, 5480–5489. [Google Scholar] [CrossRef]
11. Zhao Rongyong, Wei Bingyu, Zhu Wenjie, et al. Review of pedestrian abnormal behavior recognition methods in public places. China Safety Science Journal, 2024, 34(2):83-93. [CrossRef]
12. Wang Duorui, Du Yang, Dong Lanfang, Hu Weiming, Li Bing. Small sample learning algorithm based on feature transformation and metric network. Acta Automatica Sinica, 2024, 50(7): 1305−1314.
13. Ng S H, Platow M J. The violent turn in non-violent collective action: What happens?. Asian Journal of Social Psychology, 2024.
14. Chen Chong, Bai Shuo, Huang Lida, et al. Research on passenger flow monitoring and early warning in crowded places based on video analysis. Journal of Safety Science and Technology, 2020, 16 (04): 143-148.
15. Wang, X.; Li, Y. Edge Detection and Simulation Analysis of Multimedia Images Based on Intelligent Monitoring Robot. Informatica 2024, 48. [Google Scholar] [CrossRef]
16. Chen X, Yan B, Zhu J, et al. High-performance transformer tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(7): 8507-8523.
17. Farooq, M.U.; Saad, M.N.B.M.; Malik, A.S.; Ali, Y.S.; Khan, S.D. Motion estimation of high density crowd using fluid dynamics. Imaging Sci. J. 2020, 68, 141–155. [Google Scholar] [CrossRef]
18. Zhou Zhiqiang, Sun Riming, Guo Chenglong, et al. Hierarchical motion estimation method for spatially unstable targets under Gaussian mixture model[J/OL]. Acta Optica Sinica:1-19[2024-05-26].http://kns.cnki.net/kcms/detail/31.1252.O4.20240517.1540.029.html.
19. Zhou, Y.; Liu, C.; Ding, Y.; Yuan, D.; Yin, J.; Yang, S.-H. Crowd Descriptors and Interpretable Gathering Understanding. IEEE Trans. Multimedia 2024, 26, 8651–8664. [Google Scholar] [CrossRef]
20. Alhothali, A.; Balabid, A.; Alharthi, R.; Alzahrani, B.; Alotaibi, R.; Barnawi, A. Anomalous event detection and localization in dense crowd scenes. Multimedia Tools Appl. 2022, 82, 15673–15694. [Google Scholar] [CrossRef]
21. Nayan, N.; Sahu, S.S.; Kumar, S. Detecting anomalous crowd behavior using correlation analysis of optical flow. Signal, Image Video Process. 2019, 13, 1233–1241. [Google Scholar] [CrossRef]
22. Aziz, Z.; Bhatti, N.; Mahmood, H.; Zia, M. Video anomaly detection and localization based on appearance and motion models. Multimedia Tools Appl. 2021, 80, 1–21. [Google Scholar] [CrossRef]
23. Matkovic, F.; Ivasic-Kos, M.; Ribaric, S. A new approach to dominant motion pattern recognition at the macroscopic crowd level. Eng. Appl. Artif. Intell. 2022, 116. [Google Scholar] [CrossRef]
24. Ganga, B.; B. T., L.; K.R., V. Object detection and crowd analysis using deep learning techniques: Comprehensive review and future directions. Neurocomputing 2024, 597. [Google Scholar] [CrossRef]
25. Madan, N.; Ristea, N.-C.; Ionescu, R.T.; Nasrollahi, K.; Khan, F.S.; Moeslund, T.B.; Shah, M. Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 46, 525–542. [Google Scholar] [CrossRef]
26. Chai, L.; Liu, Y.; Liu, W.; Han, G.; He, S. CrowdGAN: Identity-Free Interactive Crowd Video Generation and Beyond. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 44, 2856–2871. [Google Scholar] [CrossRef]
27. Song, X.; Chen, K.; Li, X.; Sun, J.; Hou, B.; Cui, Y.; Zhang, B.; Xiong, G.; Wang, Z. Pedestrian Trajectory Prediction Based on Deep Convolutional LSTM Network. IEEE Trans. Intell. Transp. Syst. 2020, 22, 3285–3302. [Google Scholar] [CrossRef]
28. Huang Y, Abeliuk A, Morstatter F, et al. Anchor attention for hybrid crowd forecasts aggregation. arXiv:2003.12447, 2020.
29. He, F.; Xiang, Y.; Zhao, X.; Wang, H. Informative scene decomposition for crowd analysis, comparison and simulation guidance. ACM Trans. Graph. 2020, 39. [Google Scholar] [CrossRef]
30. Afiq A A, Zakariya M A, Saad M N, et al. A review on classifying abnormal behavior in crowd scene. Journal of Visual Communication and Image Representation, 2019, 58: 285-303.
31. Goel S, Koundal D, Nijhawan R. Learning Models in Crowd Analysis: A Review. Archives of Computational Methods in Engineering, 2024: 1-19.
32. Rajasekaran, G.; Sekar, J.R. Abnormal Crowd Behavior Detection Using Optimized Pyramidal Lucas-Kanade Technique. Intell. Autom. Soft Comput. 2023, 35, 2399–2412. [Google Scholar] [CrossRef]
33. Gündüz M Ş, Işık G. A new YOLO-based method for real-time crowd detection from video and performance analysis of YOLO models. Journal of Real-Time Image Processing, 2023, 20(1): 5.
34. Miao, Y.; Yang, J.; Alzahrani, B.; Lv, G.; Alafif, T.; Barnawi, A.; Chen, M. Abnormal Behavior Learning Based on Edge Computing toward a Crowd Monitoring System. IEEE Netw. 2022, 36, 90–96. [Google Scholar] [CrossRef]
35. Wang Kunlun, Liu Wencan, He Xiaohai, et al. A motion feature descriptor for abnormal behavior detection. Computer Science, 2020, 47(4):119-124. [CrossRef]
36. Qin Yue, Shi Yuexiang. Human behavior recognition based on two-stream network fusion and spatio-temporal convolution. Computing Technology and Automation, 2021, 40 (02): 140-147. [CrossRef]
37. Jiang, J.; Wang, X.; Gao, M.; Pan, J.; Zhao, C.; Wang, J. Abnormal behavior detection using streak flow acceleration. Appl. Intell. 2022, 52, 10632–10649. [Google Scholar] [CrossRef]
38. Clarke K, C. Cellular automata and agent-based models[M]//Handbook of regional science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2021: 1751-1766.
39. Li Changhua, Bi Chenggong, Li Zhijie. Crowd evacuation model based on improved PSO algorithm. Journal of System Simulation, 2020, 32 (06): 1000-1008. [CrossRef]
40. Chang, D.; Cui, L.; Huang, Z. A Cellular-Automaton Agent-Hybrid Model for Emergency Evacuation of People in Public Places. IEEE Access 2020, 8, 79541–79551. [Google Scholar] [CrossRef]
41. Chebi H, Dalila A, Mohamed K. Dynamic detection of abnormalities in video analysis of crowd behavior with DBSCAN and neural networks. Advances in Science, Technology and Engineering Systems Journal, 2020, 1(5): 56-63.
42. Tay N C, Connie T, Ong T S, et al. A robust abnormal behavior detection method using convolutional neural network[C]//Computational Science and Technology: 5th ICCST 2018, Kota Kinabalu, Malaysia, 29-30 August 2018. Springer Singapore, 2019: 37-47.
43. Koonce B, Koonce B. Vgg network. Convolutional Neural Networks with Swift for Tensorflow: Image Recognition and Dataset Categorization, 2021: 35-50.
44. Koonce B, Koonce B. ResNet 50. Convolutional neural networks with swift for tensorflow: image recognition and dataset categorization, 2021: 63-72.
45. Behera N K S, Sa P K, Muhammad K, et al. Large-Scale Person Re-Identification for Crowd Monitoring in Emergency. IEEE Transactions on Automation Science and Engineering, 2023.
46. Wang, S.; Pu, Z.; Li, Q.; Wang, Y. Estimating crowd density with edge intelligence based on lightweight convolutional neural networks. Expert Syst. Appl. 2022, 206. [Google Scholar] [CrossRef]
47. Dai F, Huang P, Mo Q, et al. ST-InNet: Deep spatio-temporal inception networks for traffic flow prediction in smart cities. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(10): 19782-19794.
48. Liu W, Luo W, Li Z, et al. Margin Learning Embedded Prediction for Video Anomaly Detection with A Few Anomalies[C]//IJCAI. 2019, 3: 023-3.
49. Bhuiyan, R.; Abdullah, J.; Hashim, N.; Al Farid, F.; Uddin, J. Hajj pilgrimage abnormal crowd movement monitoring using optical flow and FCNN. J. Big Data 2023, 10, 1–21. [Google Scholar] [CrossRef]
50. Garg, S.; Sharma, S.; Dhariwal, S.; Priya, W.D.; Singh, M.; Ramesh, S. Human crowd behaviour analysis based on video segmentation and classification using expectation–maximization with deep learning architectures. Multimedia Tools Appl. 2024, 1–23. [Google Scholar] [CrossRef]
51. Singh, K.; Rajora, S.; Vishwakarma, D.K.; Tripathi, G.; Kumar, S.; Walia, G.S. Crowd anomaly detection using Aggregation of Ensembles of fine-tuned ConvNets. Neurocomputing 2019, 371, 188–198. [Google Scholar] [CrossRef]
52. Sabokrou, M.; Fayyaz, M.; Fathy, M.; Moayed, Z.; Klette, R. Deep-anomaly: Fully convolutional neural network for fast anomaly detection in crowded scenes. Comput. Vis. Image Underst. 2018, 172, 88–97. [Google Scholar] [CrossRef]
53. Hu, X.; Dai, J.; Huang, Y.; Yang, H.; Zhang, L.; Chen, W.; Yang, G.; Zhang, D. A weakly supervised framework for abnormal behavior detection and localization in crowded scenes. Neurocomputing 2020, 383, 270–281. [Google Scholar] [CrossRef]
54. Wang Hongyan, Zhou Mengxing. Crowd abnormal behavior detection based on optical flow and trajectory. Journal of Jilin University (Engineering Edition), 2020, 50 (06): 2229-2237. [CrossRef]
55. Hu Xuemin, Chen Qin, Yang Li, et al. Crowd abnormal behavior detection and location based on deep spatio-temporal convolutional neural network. Application Research of Computers, 2020, 37(3):891-895. [CrossRef]
56. Gong, M.; Zeng, H.; Xie, Y.; Li, H.; Tang, Z. Local distinguishability aggrandizing network for human anomaly detection. Neural Networks 2020, 122, 364–373. [Google Scholar] [CrossRef]
57. Pinaya W H L, Vieira S, Garcia-Dias R, et al. Autoencoders[M]//Machine learning. Academic Press, 2020: 193-208.
58. Tyagi, B.; Nigam, S.; Singh, R. A Review of Deep Learning Techniques for Crowd Behavior Analysis. Arch. Comput. Methods Eng. 2022, 29, 5427–5455. [Google Scholar] [CrossRef]
59. Xu Tao, Tian Chongyang. Review of abnormal behavior detection of crowd based on deep learning. Computer Science, 2021, 48(9):125-134. [CrossRef]
60. Zhang, X.; Ma, D.; Yu, H.; Huang, Y.; Howell, P.; Stevens, B. Scene perception guided crowd anomaly detection. Neurocomputing 2020, 414, 291–302. [Google Scholar] [CrossRef]
61. Zhou, S.; Shi, R.; Wang, L. Extracting macroscopic quantities in crowd behaviour with deep learning. Phys. Scr. 2024, 99, 065213. [Google Scholar] [CrossRef]
62. Nguyen T N, Meunier J. Anomaly detection in video sequence with appearance-motion correspondence[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 1273-1283.
63. Wei H, Li K, Li H, et al. Detecting video anomaly with a stacked convolutional LSTM framework[C]//International Conference on Computer Vision Systems. Cham: Springer International Publishing, 2019: 330-342.
64. Xiao Jinsheng, Guo Haowen, Xie Honggang, et al. Probabilistic memory autoencoder network for abnormal behavior detection in surveillance videos. Journal of Software, 2023, 34(9): 4362-4377.
65. Nawaratne, R.; Alahakoon, D.; De Silva, D.; Yu, X. Spatiotemporal Anomaly Detection Using Deep Learning for Real-Time Video Surveillance. IEEE Trans. Ind. Informatics 2019, 16, 393–402. [Google Scholar] [CrossRef]
66. Roka, S.; Diwakar, M. Deep stacked denoising autoencoder for unsupervised anomaly detection in video surveillance. J. Electron. Imaging 2023, 32, 033015. [Google Scholar] [CrossRef]
67. Li, J.; Huang, Q.; Du, Y.; Zhen, X.; Chen, S.; Shao, L. Variational Abnormal Behavior Detection With Motion Consistency. IEEE Trans. Image Process. 2021, 31, 275–286. [Google Scholar] [CrossRef]
68. Ashfahani, A.; Pratama, M.; Lughofer, E.; Ong, Y.-S. DEVDAN: Deep evolving denoising autoencoder. Neurocomputing 2019, 390, 297–314. [Google Scholar] [CrossRef]
69. Wang, J.; Xia, L. Abnormal behavior detection in videos using deep learning. Clust. Comput. 2018, 22, 9229–9239. [Google Scholar] [CrossRef]
70. Wang, T.; Qiao, M.; Lin, Z.; Li, C.; Snoussi, H.; Liu, Z.; Choi, C. Generative Neural Networks for Anomaly Detection in Crowded Scenes. IEEE Trans. Inf. Forensics Secur. 2018, 14, 1390–1399. [Google Scholar] [CrossRef]
71. Lü Chengkan, Shen Fei, Zhang Zhengtao, et al. Review of the Research Status of Image Anomaly Detection. Acta Automatica Sinica, 2022, 48(6):1402-1428. [CrossRef]
72. Song, H.; Sun, C.; Wu, X.; Chen, M.; Jia, Y. Learning Normal Patterns via Adversarial Attention-Based Autoencoder for Abnormal Event Detection in Videos. IEEE Trans. Multimedia 2019, 22, 2138–2148. [Google Scholar] [CrossRef]
73. Chen, D.; Yue, L.; Chang, X.; Xu, M.; Jia, T. NM-GAN: Noise-modulated generative adversarial network for video anomaly detection. Pattern Recognit. 2021, 116. [Google Scholar] [CrossRef]
74. Tang, Y.; Zhao, L.; Zhang, S.; Gong, C.; Li, G.; Yang, J. Integrating prediction and reconstruction for anomaly detection. Pattern Recognit. Lett. 2019, 129, 123–130. [Google Scholar] [CrossRef]
75. Lee S, Kim H G, Ro Y M. BMAN: Bidirectional multi-scale aggregation networks for abnormal event detection. IEEE Transactions on Image Processing, 2019, 29: 2395-2408.
76. Schlegl T, Seeböck P, Waldstein S M, et al. f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks. Medical image analysis, 2019, 54: 30-44.
77. Meng B, Wang L, Li D W. Detection Method for Crowd Abnormal Behavior Based on Long Short-Term Memory Network[C]//Advances in Intelligent Information Hiding and Multimedia Signal Processing: Proceeding of the 16th International Conference on IIHMSP in conjunction with the 13th international conference on FITAT, November 5-7, 2020, Ho Chi Minh City, Vietnam, Volume 1. Springer Singapore, 2021: 305-313.
78. Wu Guangli, Guo Zhenzhou, Li Leiting, et al. Video Anomaly Event Detection Fusing FCN and LSTM. Journal of Shanghai Jiao Tong University, 2021, 55(5): 607.
79. Sabih, M.; Vishwakarma, D.K. Crowd anomaly detection with LSTMs using optical features and domain knowledge for improved inferring. Vis. Comput. 2021, 38, 1719–1730. [Google Scholar] [CrossRef]
80. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Advances in Neural Information Processing Systems, 2017, 30.
81. Ye Z, Li Y, Cui Z, et al. Unsupervised Video Anomaly Detection with Self-Attention Based Feature Aggregating[C]//2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2023: 3551-3556.
82. Zhang Hongmin, Fang Xiaobing, Zhuang Xu. Video Human Abnormal Behavior Detection Model of Autoencoder Fusing Attention Mechanism. Laser Journal, 2023, 44 (02): 69-75. [CrossRef]
83. Zhang J, Qi X, Ji G. Self Attention Based Bi-Directional Long Short-Term Memory Auto Encoder for Video Anomaly Detection[C]//2021 Ninth International Conference on Advanced Cloud and Big Data (CBD). IEEE, 2022: 107-112.
84. Zhang, W.; Wang, G.; Huang, M.; Wang, H.; Wen, S. Generative Adversarial Networks for Abnormal Event Detection in Videos Based on Self-Attention Mechanism. IEEE Access 2021, 9, 124847–124860. [Google Scholar] [CrossRef]
85. Singh, R.; Sethi, A.; Saini, K.; Saurav, S.; Tiwari, A.; Singh, S. Attention-guided generator with dual discriminator GAN for real-time video anomaly detection. Eng. Appl. Artif. Intell. 2024, 131. [Google Scholar] [CrossRef]
86. Sharma, V.K.; Mir, R.N.; Singh, C. Scale-aware CNN for crowd density estimation and crowd behavior analysis. Comput. Electr. Eng. 2023, 106. [Google Scholar] [CrossRef]
87. Zhao, R.; Wang, Y.; Jia, P.; Zhu, W.; Li, C.; Ma, Y.; Li, M. Abnormal Behavior Detection Based on Dynamic Pedestrian Centroid Model: Case Study on U-Turn and Fall-Down. IEEE Trans. Intell. Transp. Syst. 2023, 24, 8066–8078. [Google Scholar] [CrossRef]
88. UCSD Anomaly Detection Dataset, http://www.svcl.ucsd.edu/projects/anomaly/dataset.htm.
89. UMN Crowd Dataset, http://mha.cs.umn.edu/projevents.shtml#crowd.
90. Cao, C.; Lu, Y.; Zhang, Y. Context Recovery and Knowledge Retrieval: A Novel Two-Stream Framework for Video Anomaly Detection. IEEE Trans. Image Process. 2024, 33, 1810–1825. [Google Scholar] [CrossRef]
91. Wang, L.; Wang, X.; Li, M.; Liu, F. Anomaly Detection Method Based on Temporal Spatial Information Enhancement. Meas. Sci. Technol. 2023, 35. [Google Scholar] [CrossRef]