1. Introduction

• We implement SPT which gives a wider receptive field than standard tokenization to embed more spatial optical flow information into visual tokens for the training phase.

• We employ LSA which impels the attention to work locally by forcing each token to concentrate more on tokens with large relation to itself, to enable the network to pay more attention to visual tokens that contain important expression motion information.

• We incorporate both SPT as well as LSA into the backbone Swin Transformer to enable it to be trained from scratch on small-size expression datasets and our study demonstrates the effectiveness of the Transformer-based deep learning approach by outperforming the MEGC 2022 spotting baseline approach and achieving a comparably competent outcome in the MEGC 2021 spotting task.

2. Materials and Methods

2.3. SL-Swin

For the sake of spotting expression by training from scratch on small-size expression datasets, we propose SL-Swin with these further considerations: (1) The backbone of the network is the Swin Transformer [27] which could effectively consider the local and global features of the expression optical flow features; (2) Shifted Patch Tokenization (SPT) [26] and Locality Self-Attention (LSA) [26] are applied to the backbone to enable the network to be trained from scratch even on small-size expression datasets. (3) A head module is added to predict a score indicating the probability of a frame being within an expression interval.

Figure 2 illustrated an overview of the model SL-Swin-T, which is based on the tiny version of the Swin Transformer (Swin-T). SL means both SPT as well as LSA are applied to the model. To start with, the SPT module in “Stage 1” splits the input optical flow features into non-overlapping patches which are treated as “visual tokens”. In our approach, the patch size p is 6, and the output is projected to dimension C by a linear embedding layer in SPT. Next, the tokens go through several Transformer blocks with LSA (L Swin Transformer blocks) which keep the resolution of tokens at $(\frac{H}{6}\times \frac{W}{6})$. and together with the SPT are referred to as “Stage 1”. The H and W are the width and weight of the pre-processed optical flow features which is put into the model.

The number of tokens is decreased by patch merging layers in the following “Stages” as the network gets deeper because the backbone Swin Transformer is designed to build hierarchical feature maps. The patch merging layer decreases the number of tokens by a multiple of $2\times 2=4$ ($2\times$ downsampling of resolution) by concatenating the tokens of each group of $2\times 2$ adjacent patches. Then, a linear layer within the patch merging layer is applied to project the downsampled $4C$-dimensional concatenated features to the $4C$-dimensional output. Afterwards, feature transformation is conducted by several L Swin Transformer blocks (L means LSA is applied) which maintain the resolution at $\frac{H}{12}\times \frac{W}{12}$. This first combination of the patch merging layer and several L Swin Transformer blocks is denoted as “Stage 2”. The procedure is repeated twice, as “Stage 3” and “Stage 4”, with output resolutions of $\frac{H}{24}\times \frac{W}{24}$ and $\frac{H}{48}\times \frac{W}{48}$, respectively. In the end, a head that acts as a regression module which consists of the normalization and the MLP is applied to predict a score indicating the probability of a frame being within an expression interval.

2.3.1. Swin Transformer

Facial expressions could be divided into individual muscle movement components, known as Action Units (AUs) [5]. As the experiment described in [32], a single macro- or micro-expression may have more than one AU with high intensity. Consequently, to identify whether a macro- or micro-expression appears or not, the model must take the local features, global features, and the relationships among local features from various input portions into consideration.

Inspired by the attention mechanism [17] in the field of Natural Language Processing (NLP) and the ViT [18] which applies attention to the field of Computer Vision (CV), we use a deep learning model called SL-Swin which uses Swin Transformer [27] as the backbone. The Swin Transformer is built in hierarchical architecture and the transformer representation is computed with shifted windows. The standard Transformer architecture conducts global self-attention which leads to quadratic computation complexity in respect of the number of tokens because the relationships between a token and all other tokens are computed. For efficient modeling, the Swin Transformer computes self-attention within windows that evenly partition the tokens into non-overlapping parts. To address the drawback of lacking connections across windows in the window based self-attention module, the shifted window partitioning approach which shifts the window and computes the self-attention within the new windows that cross the boundaries of the non-overlapping windows is proposed in consecutive Swin Transformer blocks. In the computation of every consecutive Swin Transformer block, the window based multi-head self-attention using regular window partitioning configurations (W-MSA) and window based multi-head self-attention using shifted window partitioning configurations (SW-MSA) are used in pairs. The shifted windowing scheme which includes regular and shifted window partitioning configurations shows efficient modeling power by confining self-attention computation to non-overlapping windows while harnessing cross-window connections. Therefore, the network could effectively take local features, global features, and the relation among local features from different parts of the input optical flow features into consideration.

Whereas the models based on Transformers such as ViT and Swin Transformer require a large amount of training data or require pre-training on a large-size dataset to obtain high performance. The dataset for micro-expression spotting is relatively small, which may limit the performance of the model based on the Transformer. To enable the network to perform well on comparatively small-scale expression datasets, we implement SPT which gives a wider receptive field to the model than standard tokenization by embedding more spatial information in visual tokens. And we employ LSA which enables the network to pay more attention to visual tokens that contain important motion information. The details of SPT and LSA are demonstrated below.

2.3.2. Shifted Patch Tokenization

The Shifted Patch Tokenization (SPT) outputs the tensor with the same shape as the original Patch Embedding Layer of the Swin Transformer or Vision Transformer. Therefore, we use the SPT as a Patch Embedding Layer in our approach. The SPT is the head component of the SL-Swin model, which means the pre-processed optical flow features will be processed by the SPT first when the features are fed into the model. The following describes the overall formulation of the SPT and how we implement the SPT as a Patch Embedding Layer.

Firstly, the shifting strategy is applied to shift each input pre-processed optical flow features by half the patch size in four diagonal directions which are left-up, right-up, left-down, and right-down. Next, the shifted features are concatenated with the original input pre-processed optical flow features after being cropped to the same size as the input. Then, the concatenated features $[x,x_s^1,x_s^2,x_s^3,x_s^4]$ are divided into non-overlapping patches and the patches are flattened to a sequence of vectors, which are formulated as follows:

$$DF\left(\left[x,x_s^1,x_s^2,x_s^3,x_s^4\right]\right)=\left[x_p^1;x_p^2;\dots ;x_p^N\right]$$

(3)

Where x is the original pre-processed optical flow features and $x_s^1$, $x_s^2$, $x_s^3$, $x_s^4$ are the cropped features shifted in left-up, right-up, left-down, and right-down directions. $x_p^i\in {\mathbb{R}}^{P^2\times C}$ is the i-th flattened vector. p stands for the patch size and $N=\frac{HW}{p^2}$ stands for the number of patches. $DP$ stands for the dividing and flattening process.

Afterwards, visual tokens ($VT$) are obtained through layer normalization ($LN$) and projected by a linear layer ($LL$). The whole process is formulated as:

$$VT\left(x\right)=LL\left(LN\left(\left[x_p^1;x_p^2;\dots ;x_p^N\right]\right)\right)$$

(4)

In order to use SPT as a Patch Embedding Layer, we add a positional embedding variable to the output of SPT. The whole process is formulated as:

$$V{T}_{pe}\left(x\right)=VT\left(x\right)+E_{pos}$$

(5)

Where $E_{pos}$ is the learnable positional embedding variable and $V{T}_{pe}\left(x\right)$ is the ultimate output that will be processed by the rest of the model.

In all, the SPT in our approach could be seen as the combination of the Patch Partition and the Linear Embedding in “Stage 1” of the original Swin Transformer architecture.

2.3.3. Locality Self-Attention

The diagonal masking and learnable temperature scaling consist of the core of the Locality Self-Attention Mechanism (LSA). Figure 3a demonstrates the difference between the standard self-attention mechanism and the locality self-attention used in the SL-Swin model. Figure 3b also shows how LSA is applied in the successive Swin Transformer blocks to form the L Swin Transformer block. The W-MLSA and SW-MLSA in the two successive Swin Transformer blocks of Figure 3b denote window based multi-head locality self-attention using regular and shifted window partitioning configurations, respectively. The L indicates that the LSA is applied.

The standard self-attention computation of general ViTs operates as follows. In the beginning, the Query, Key, and Value are obtained by applying a learnable linear projection to each token. Next, calculate the similarity matrix R which represents the relation between tokens is calculated through the dot product operation of Query and Key. The diagonal and the off-diagonal components of R represent self-token and inter-token relations, respectively:

$$R=Q{K}^T$$

(6)

Here, Q and K denote learnable linear projections for Query and Key. Afterwards, the diagonal masking forces $-\infty$ on diagonal components of R to emphasize the inter-token relations by basically excluding self-token relations from the following computation. This enforce the model to concentrate more on other tokens rather than its inter-tokens. The diagonal masking is formulated as:

$$R^M=\left\{\begin{array}{c}{R}_{i,j}\left(i\ne j\right)\\ -\infty \left(i=j\right)\end{array}\right.$$

(7)

Where $R^M$ represents the masked similarity matrix. i and j indicate the row and column index of the similarity matrix R.

After diagonal masking, the learnable temperature scaling is applied for allowing the model to determine the softmax temperature by itself during the learning phase. As the attention mechanism is used in the SW-MSA and SW-MSA of the backbone Swin Transformer, we also include a relative position bias B for sustaining the backbone architecture. Finally, the attention score matrix is attained through the softmax operation. And the self-attention matrix is acquired by the dot product of the attention score matrix and Value:

$$Attention\left(Q,K,V\right)=softmax\left(\frac{R^M}{\tau }+B\right)V$$

(8)

Where V is the learnable linear projection of Value, and $\tau$ is the learnable temperature.

Figure 3. (a) The comparison between the standard self-attention mechanism and the locality self-attention mechanism; (b) two successive L Swin Transformer Blocks. W-MLSA and SW-MLSA are multi-head locality self-attention modules with regular and shifted windowing configurations, respectively.

Figure 3. (a) The comparison between the standard self-attention mechanism and the locality self-attention mechanism; (b) two successive L Swin Transformer Blocks. W-MLSA and SW-MLSA are multi-head locality self-attention modules with regular and shifted windowing configurations, respectively.

3. Results

3.4. Permormances

The results of our approach in the MEGC 2022 spotting task are shown in Table 1. And Table 2 compares the results of our approach and other approaches which are categorized into traditional approaches and deep learning approaches. The discussion about the results as well as the detail of the MEGC 2021 spotting task is shown in the Discussion section.

3.4.1. MEGC 2022 Spotting Task

Table 1. Performance comparison of our approach in the MEGC 2022 spotting task.

Table 1. Performance comparison of our approach in the MEGC 2022 spotting task.

Approaches	CAS(ME)3 Challenge			SAMM Challenge			Overall
	Precision	Recall	F1-score	Precision	Recall	F1-score	Precision	Recall	F1-score
Baseline [15]	0.4000	0.1111	0.1739	0.0845	0.1935	0.1176	0.1235	0.1493	0.1351
Swin-T	0.1521	0.1944	0.1707	0.6380	0.0967	0.0769	0.1075	0.1492	0.1250
Ours	0.1944	0.1944	0.1944	0.0689	0.1290	0.0898	0.1170	0.1641	0.1366

3.4.2. MEGC 2021 Spotting Task

Table 2. Performance comparison of our approach against the others in the MEGC 2021 spotting task (F1-score)

Table 2. Performance comparison of our approach against the others in the MEGC 2021 spotting task (F1-score)

Dataset	CAS(ME)2			SAMM Long Videos
Approaches	Macro	Micro	Overall	Macro	Micro	Overall
Traditional Approaches
He et al. [38]	0.1196	0.0082	0.0376	0.0629	0.0364	0.0445
Zhang et al. [8]	0.2131	0.0547	0.1403	0.0725	0.1331	0.0999
He et al. [39]	0.3782	0.1965	0.3436	0.4149	0.2162	0.3638
Deep Learning Approaches
Baseline [15]	0.2145	0.0714	0.1675	0.1595	0.0466	0.1084
Yand et al. [7]	0.2599	0.0339	0.2118	0.3553	0.1155	0.2736
Yu et al. [40]	0.3800	0.0630	0.3270	0.3360	0.2180	0.2900
Liong et al. [25]	0.2410	0.1173	0.2022	0.2169	0.1520	0.1881
Liong et al. [41]	0.4104	0.0808	0.3250	0.2810	0.1310	0.2380
Ours	0.2236	0.0879	0.1824	0.1675	0.1044	0.1357

4. Discussion

4.2. MEGC 2021 Spotting Task

Table 3. Detail of the SL-Swin-T in the MEGC 2021 spotting task

Table 3. Detail of the SL-Swin-T in the MEGC 2021 spotting task

Dataset	CAS(ME)2			SAMM Long Videos			Overall
Expression	MaE	ME	Overall	MaE	ME	Overall
Total	298	57	355	331	159	490	845
TP	70	12	82	52	33	85	167
FP	258	204	462	238	440	678	1140
FN	228	45	273	279	126	405	678
Precision	0.2134	0.0556	0.1507	0.1793	0.0698	0.1114	0.1278
Recall	0.2349	0.2105	0.2310	0.1571	0.2075	0.1735	0.1976
F1-score	0.2236	0.0879	0.1824	0.1675	0.1044	0.1357	0.1552

For comparison, Table 2 shows the results of our approach against other approaches which are categorized into traditional and deep learning approaches. Overall, our approach outperforms the baseline and proves that the Transformer-based model could perform competitively compared to the model based on Convolution Neural Networks.

Our approach outperforms all traditional approaches except for the state-of-the-art approach proposed by He et al. [39]. Among the deep learning approaches, our approach remains competitive especially on ME spotting for CAS(ME)² dataset with the F1-score of 0.0879, only behind the approach proposed by Liong et al. [25] which spots a larger amount of TPs.

4.3. Ablation Studies

We conduct experiments to provide a thorough examination of our approach, focusing on network construction, labeling function, and feature sizes. Experiments are conducted on the CAS(ME)² dataset, using similar settings as the MEGC 2021 spotting task.

4.3.1. Network Architecture

To evaluate the effectiveness of the self-attention mechanism, SPT, and LSA, we conducted a similar experiment setting using different combinations of network architecture, SPT, and LSA. S indicates that SPT is applied to the network, L indicates that LSA is applied to the network, and SL indicates that both SPT and LSA are applied. Table 4 displays the experimental results of the various network architectures. According to our findings, SPT and LSA collectively enhance the network’s performance on small-size datasets, specifically ME spotting of CAS(ME)². It is worth noting that the Swin-T model or the models that are based on it are only approximately $0.25\times$ the model size and computational complexity of the ViT-B and the SL-ViT-B.

Table 4. Performance comparison of our model (SL-Swin-T) against other Transformer-based models (F1-score)

Table 4. Performance comparison of our model (SL-Swin-T) against other Transformer-based models (F1-score)

Network Architecture	CAS(ME)2
	MaE	ME	Overall
ViT-B	0.2125	0.0158	0.1415
SL-ViT-B	0.2071	0.0940	0.1738
Swin-T	0.2110	0.0783	0.1663
S-Swin-T	0.2351	0.0749	0.1685
L-Swin-T	0.2378	0.0493	0.1765
SL-Swin-T	0.2236	0.0879	0.1824

4.3.2. Labeling

An ablation study is carried out on the original labeling and pseudo-labeling functions to investigate their impact on spotting when modeling the task as a regression problem. We set the parameter $t=0.60$ and compare the result on the SL-Swin-T model using original labeling and pseudo labeling separately. The results are shown in Table 5. We observe that applying pseudo labeling reduces the amount of False Positives (FPs), thus enhancing precision and F1-score, particularly in the overall analysis.

Table 5. Performance comparison of pseudo labeling and original labeling.

Table 5. Performance comparison of pseudo labeling and original labeling.

Dataset		CAS(ME)2
Expression	Labeling	TP	FP	FN	Precision	Recall	F1-score
MaE	Original	71	264	227	0.2119	0.2383	0.2243
	Pseudo	70	258	228	0.2134	0.2349	0.2236
ME	Original	14	262	43	0.0507	0.2456	0.0841
	Pseudo	12	204	45	0.0556	0.2105	0.0879
Overall	Original	85	526	270	0.1391	0.2394	0.1760
	Pseudo	82	462	273	0.1507	0.2310	0.1824

4.3.3. Features Size

In our approach, the SL-Swin-T model takes optical flow features $(u,v,|ϵ\left|\right)$ of size $(42,42,3)$ as input. Due to the Patch Merging in each stage of the model, the feature map is downsampled by a rate of 2, leading to only $\frac{42}{48}\times \frac{42}{48}=0.875\times 0.875$ pixels from the original optical flow features in “Stage 4”. To accommodate the windowing configuration, we apply padding when the feature map size is not an integer multiple of the window size $M=7$. With the hierarchical architecture and self-attention computation within windows, the Swin Transformer has linear computational complexity to image size. This makes the Swin Transformer suitable for processing high-resolution images, in contrast to previous Transformer based architectures which produce feature maps of a single resolution and have quadratic complexity. Hence, we double the hyper-parameters in feature extraction and pre-processing, resulting in the size of $(u,v,|ϵ\left|\right)$ being $(84,84,3)$ and allowing $\frac{84}{48}\times \frac{84}{48}=1.75\times 1.75$ pixels from the original optical flow features in “Stage 4” of the model. The hyper-parameters for the SL-Swin-T model and training configuration remain unchanged and the spotting parameter t is also set to 0.60 for comparison. Experimental results for ME spotting are presented in Table 6, demonstrating that the model performs better across all indicators, with particularly strong results in terms of the F1-score.

Table 6. Performance comparison of optical flow features of size.

Table 6. Performance comparison of optical flow features of size.

Dataset		CAS(ME)2
Expression	Features of Size	TP	FP	FN	Precision	Recall	F1-score
ME	(42, 42, 3)	12	204	45	0.0556	0.2105	0.0879
	(84, 84, 3)	13	190	44	0.0640	0.2281	0.1000

5. Conclusions

Abbreviations

References

1. Yan, W.J.; Wu, Q.; Liang, J.; Chen, Y.H.; Fu, X. How Fast are the Leaked Facial Expressions: The Duration of Micro-Expressions. Journal of Nonverbal Behavior 2013, 37, 217–230. [Google Scholar] [CrossRef]
2. Valstar, M.F.; Pantic, M. Fully Automatic Recognition of the Temporal Phases of Facial Actions. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 2012, 42, 28–43. [Google Scholar] [CrossRef] [PubMed]
3. Ben, X.; Ren, Y.; Zhang, J.; Wang, S.; Kpalma, K.; Meng, W.; Liu, Y. Video-Based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022, 44, 5826–5846. [Google Scholar] [CrossRef] [PubMed]
4. Wang, S.; Wu, S.; Qian, X.; Li, J.; Fu, X. A main directional maximal difference analysis for spotting facial movements from long-term videos. NEUROCOMPUTING 2017, 230, 382–389. [Google Scholar] [CrossRef]
5. Yang, B.; Wu, J.; Zhou, Z.; Komiya, M.; Kishimoto, K.; Xu, J.; Nonaka, K.; Horiuchi, T.; Komorita, S.; Hattori, G.; et al. Facial Action Unit-Based Deep Learning Framework for Spotting Macro- and Micro-Expressions in Long Video Sequences. In Proceedings of the Proceedings of the 29th ACM International Conference on Multimedia; Association for Computing Machinery: New York, NY, USA, 2021; MM ’21, pp. 4794–4798. event-place: Virtual Event, China. [CrossRef]
6. Davison, A.K.; Yap, M.H.; Lansley, C. Micro-Facial Movement Detection Using Individualised Baselines and Histogram-Based Descriptors. In Proceedings of the 2015 IEEE International Conference on Systems, Man, and Cybernetics; 2015; pp. 1864–1869. [Google Scholar] [CrossRef]
7. Duque, C.A.; Alata, O.; Emonet, R.; Legrand, A.C.; Konik, H. Micro-Expression Spotting Using the Riesz Pyramid. In Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV); 2018; pp. 66–74. [Google Scholar] [CrossRef]
8. Zhang, L.W.; Li, J.; Wang, S.J.; Duan, X.H.; Yan, W.J.; Xie, H.Y.; Huang, S.C. Spatio-temporal fusion for Macro- and Micro-expression Spotting in Long Video Sequences. In Proceedings of the 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020); 2020; pp. 734–741. [Google Scholar] [CrossRef]
9. LI, J.; Wang, S.J.; Yap, M.H.; See, J.; Hong, X.; Li, X. MEGC2020 - The Third Facial Micro-Expression Grand Challenge. In Proceedings of the 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020); 2020; pp. 777–780. [Google Scholar] [CrossRef]
10. Yu, J.; Cai, Z.; Liu, Z.; Xie, G.; He, P. Facial Expression Spotting Based on Optical Flow Features. In Proceedings of the Proceedings of the 30th ACM International Conference on Multimedia; Association for Computing Machinery: New York, NY, USA, 2022; MM ’22, pp. 7205–7209. event-place: Lisboa, Portugal. [CrossRef]
11. Zhang, Z.; Chen, T.; Meng, H.; Liu, G.; Fu, X. SMEConvNet: A Convolutional Neural Network for Spotting Spontaneous Facial Micro-Expression From Long Videos. IEEE Access 2018, 6, 71143–71151. [Google Scholar] [CrossRef]
12. Pan, H.; Xie, L.; Wang, Z. Local Bilinear Convolutional Neural Network for Spotting Macro- and Micro-expression Intervals in Long Video Sequences. In Proceedings of the 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020); 2020; pp. 749–753. [Google Scholar] [CrossRef]
13. Li, J.; Yap, M.H.; Cheng, W.H.; See, J.; Hong, X.; Li, X.; Wang, S.J. FME’21: 1st Workshop on Facial Micro-Expression: Advanced Techniques for Facial Expressions Generation and Spotting. In Proceedings of the Proceedings of the 29th ACM International Conference on Multimedia; Association for Computing Machinery: New York, NY, USA, 2021; MM ’21, pp. 5700–5701. event-place: Virtual Event, China. [CrossRef]
14. Li, J.; Yap, M.H.; Cheng, W.H.; See, J.; Hong, X.; Li, X.; Wang, S.J.; Davison, A.K.; Li, Y.; Dong, Z. MEGC2022: ACM Multimedia 2022 Micro-Expression Grand Challenge. In Proceedings of the Proceedings of the 30th ACM International Conference on Multimedia; Association for Computing Machinery: New York, NY, USA, 2022; MM ’22, pp. 7170–7174. event-place: Lisboa, Portugal. [CrossRef]
15. Yap, C.H.; Yap, M.H.; Davison, A.; Kendrick, C.; Li, J.; Wang, S.J.; Cunningham, R. 3D-CNN for Facial Micro- and Macro-Expression Spotting on Long Video Sequences Using Temporal Oriented Reference Frame. In Proceedings of the Proceedings of the 30th ACM International Conference on Multimedia; Association for Computing Machinery: New York, NY, USA, 2022; MM ’22, pp. 7016–7020. event-place: Lisboa, Portugal. [CrossRef]
16. Verburg, M.; Menkovski, V. Micro-expression detection in long videos using optical flow and recurrent neural networks. In Proceedings of the 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), 2019, pp. 1–6. [CrossRef]
17. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, \.; Polosukhin, I. Attention is All You Need. In Proceedings of the Proceedings of the 31st International Conference on Neural Information Processing Systems; Curran Associates Inc.: Red Hook, NY, USA, 2017; NIPS’17, pp. 6000–6010. event-place: Long Beach, California, USA. [CrossRef]
18. Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv 2020, p. 21 pp. Type: Journal Paper. [CrossRef]
19. J. Deng.; W. Dong.; R. Socher.; L. -J. Li.; Kai Li.; Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255. Journal Abbreviation: 2009 IEEE Conference on Computer Vision and Pattern Recognition. [CrossRef]
20. Pan, H.; Xie, L.; Wang, Z. Spatio-temporal convolutional emotional attention network for spotting macro- and micro-expression intervals in long video sequences. Pattern Recognition Letters 2022, 162, 89–96. [Google Scholar] [CrossRef]
21. Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers); Association for Computational Linguistics: Minneapolis, Minnesota, 2019; pp. 4171–4186. [CrossRef]
22. Zhou, Y.; Song, Y.; Chen, L.; Chen, Y.; Ben, X.; Cao, Y. A Novel Micro-Expression Detection Algorithm Based on BERT and 3DCNN. Image Vision Comput. 2022, 119, Butterworth–Heinemann. [Google Scholar] [CrossRef]
23. Guo, X.; Zhang, X.; Li, L.; Xia, Z. Micro-expression spotting with multi-scale local transformer in long videos. Pattern Recognition Letters 2023, 168, 146–152. [Google Scholar] [CrossRef]
24. Liong, S.T.; Gan, Y.S.; See, J.; Khor, H.Q.; Huang, Y.C. Shallow Triple Stream Three-dimensional CNN (STSTNet) for Micro-expression Recognition. In Proceedings of the 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), 2019, pp. 1–5. [CrossRef]
25. G. -B. Liong.; J. See.; L. -K. Wong. Shallow Optical Flow Three-Stream CNN For Macro- And Micro-Expression Spotting From Long Videos. In Proceedings of the 2021 IEEE International Conference on Image Processing (ICIP), 2021, pp. 2643–2647. Journal Abbreviation: 2021 IEEE International Conference on Image Processing (ICIP). [CrossRef]
26. Lee, S.; Lee, S.; Song, B. Improving Vision Transformers to Learn Small-Size Dataset From Scratch. IEEE ACCESS 2022, 10, 123212–123224. [Google Scholar] [CrossRef]
27. Z. Liu.; Y. Lin.; Y. Cao.; H. Hu.; Y. Wei.; Z. Zhang.; S. Lin.; B. Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 9992–10002. Journal Abbreviation: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). [CrossRef]
28. Moilanen, A.; Zhao, G.; Pietikäinen, M. Spotting Rapid Facial Movements from Videos Using Appearance-Based Feature Difference Analysis. In Proceedings of the 2014 22nd International Conference on Pattern Recognition, 2014, pp. 1722–1727. [CrossRef]
29. Zhao, Y.; Tong, X.; Zhu, Z.; Sheng, J.; Dai, L.; Xu, L.; Xia, X.; Jiang, Y.; Li, J. Rethinking Optical Flow Methods for Micro-Expression Spotting. In Proceedings of the Proceedings of the 30th ACM International Conference on Multimedia; Association for Computing Machinery: New York, NY, USA, 2022; MM ’22, pp. 7175–7179. event-place: Lisboa, Portugal. [CrossRef]
30. Shreve, M.; Brizzi, J.; Fefilatyev, S.; Luguev, T.; Goldgof, D.; Sarkar, S. Automatic expression spotting in videos. Image and Vision Computing 2014, 32, 476–486. [Google Scholar] [CrossRef]
31. Liong, S.T.; See, J.; Wong, K.; Phan, R.C.W. Automatic Micro-expression Recognition from Long Video Using a Single Spotted Apex. In Proceedings of the Computer Vision – ACCV 2016 Workshops; Chen, C.S.; Lu, J.; Ma, K.K., Eds.; Springer International Publishing: Cham, 2017; pp. 345–360. [CrossRef]
32. Yap, C.H.; Kendrick, C.; Yap, M.H. SAMM Long Videos: A Spontaneous Facial Micro- and Macro-Expressions Dataset. In Proceedings of the 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), 2020, pp. 771–776. [CrossRef]
33. Li, J.; Dong, Z.; Lu, S.; Wang, S.J.; Yan, W.J.; Ma, Y.; Liu, Y.; Huang, C.; Fu, X. CAS(ME)3: A Third Generation Facial Spontaneous Micro-Expression Database With Depth Information and High Ecological Validity. IEEE Transactions on Pattern Analysis and Machine Intelligence 2023, 45, 2782–2800. [Google Scholar] [CrossRef] [PubMed]
34. Davison, A.K.; Lansley, C.; Costen, N.; Tan, K.; Yap, M.H. SAMM: A Spontaneous Micro-Facial Movement Dataset. IEEE Transactions on Affective Computing 2018, 9, 116–129. [Google Scholar] [CrossRef]
35. Qu, F.; Wang, S.J.; Yan, W.J.; Li, H.; Wu, S.; Fu, X. CAS(ME)$⌃2$ : A Database for Spontaneous Macro-Expression and Micro-Expression Spotting and Recognition. IEEE Transactions on Affective Computing 2018, 9, 424–436. [Google Scholar] [CrossRef]
36. Davison, A.; Merghani, W.; Moi Hoon Yap. Objective classes for micro-facial expression recognition. Journal of Imaging 2018, 4, 119–13. [Google Scholar] [CrossRef]
37. Yan, W.J.; Li, X.; Wang, S.J.; Zhao, G.; Liu, Y.J.; Chen, Y.H.; Fu, X. CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation. PLOS ONE 2014, 9, e86041. [Google Scholar] [CrossRef] [PubMed]
38. He, Y.; Wang, S.; Li, J.; Yap, M. Spotting Macro- and Micro-expression Intervals in Long Video Sequences. In Proceedings of the Chinese Academy of Sciences; Struc, V.; GomezFernandez, F., Eds., 2020, pp. 742–748. [CrossRef]
39. He, Y.; Xu, Z.; Ma, L.; Li, H. Micro-expression spotting based on optical flow features. Pattern Recognition Letters 2022, 163, 57–64. [Google Scholar] [CrossRef]
40. Yu, W.W.; Jiang, J.; Li, Y.J. LSSNet: A Two-Stream Convolutional Neural Network for Spotting Macro- and Micro-Expression in Long Videos. In Proceedings of the Proceedings of the 29th ACM International Conference on Multimedia; Association for Computing Machinery: New York, NY, USA, 2021; MM ’21, pp. 4745–4749. event-place: Virtual Event, China. [CrossRef]
41. Liong, G.B.; Liong, S.T.; See, J.; Chan, C.S. MTSN: A Multi-Temporal Stream Network for Spotting Facial Macro- and Micro-Expression with Hard and Soft Pseudo-Labels. In Proceedings of the Proceedings of the 2nd Workshop on Facial Micro-Expression: Advanced Techniques for Multi-Modal Facial Expression Analysis; Association for Computing Machinery: New York, NY, USA, 2022; FME ’22, pp. 3–10. event-place: Lisboa, Portugal. [CrossRef]