1. Introduction

2. Background

3. Related Work

4. Method

4.3. MEVD Model

In this section, we will introduce the model structure of MEVD. The MEVD model consists of three modules: the Surface Feature Encoder (SFE), the Dual Branch Transformer-CNN Encoder and the Deep Residual Shrinkage Network (DRSN).

4.3.1. Surface Feature Encoder

We used the word2vec algorithm, which can convert tokens to vectors, but by converting code directly to vectors as input, semantic features within the code may be ignored. To solve this problem, we designed SFE, which can enhance the relevance of global semantics within the features and suppress the influence of irrelevant information, thus preserving richer syntactic semantic information. The detailed design of the SFE module is shown in Figure 3.

SFE leverages three design strategies to enhance semantic features: a gating mechanism, dilated convolutions, and depth-wise convolutions. Gating mechanisms are typically used to control the flow of information by enhancing attention to critical information. Specifically, the gating mechanism is formulated as the element-wise product of two parallel paths of linear transformation layers. One path is activated by the non-linear activation function GELU, while the other path applies the reverse operation to the vector. The outputs of these two paths are then subjected to element-wise multiplication, allowing information from one path to modulate the strength of the other, thereby enhancing focus on important semantic details and reducing the impact of irrelevant variables, On the other hand, the SFE employs dilated convolutions and depth-wise convolutions. Dilated convolutions extend the receptive field to cover a wider range of input information, helping to capture different scales and structural features within the code. Meanwhile, depth-wise convolutions focus on the local context and detail information in the input data. They efficiently capture local interdependencies between different channels in the input data, helping the SFE to enrich the semantic information within the features. Given the input vector (feature matrix) $\mathbf{X}\in {\mathbb{R}}^{L\times D}$, where L represents length of the token sequence, and D represents the dimension of the token embedding. SFE is the equation:

$$\mathrm{G}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{n}\mathrm{g}\left(\mathbf{X}\right)=\varphi \left(W_{de}^1{W}_{di}^1\right(\mathbf{X}\left)\right)\odot {Reverse(W}_{de}^2{W}_{di}^2\left(\mathbf{X}\right))$$

(1)

$$\widehat{\mathbf{X}}=LN\left(\mathrm{Gating} \left(\mathbf{X}\right)\right)+\mathbf{X}$$

(2)

Where $w_{di}^{(\cdot )}$ is the 3×3 dilated convolution and $W_{de}^{(\cdot )}$ is the 3×3 depth-wise convolution. where ⊙ denotes element-wise multiplication, $\varphi$ represents the GELU non-linearity, and LN is the layer normalization. Reverse denotes the reversal of rows in a feature matrix, e.g. swapping the first row with the last. In summary, the design of the SFE allows each level to be concerned with details that are complementary to other levels, focusing on enriching features with code semantic context information.

4.3.2. Dual-Branch Encoder

The Dual Branch Encoder consists of two parallel encoders: the Base Transformer Encoder (BTE) and the Detail CNN Encoder (DCE). These two modules are used to further extract vector features after the SFE. The BTE is used to effectively capture global structural information, allowing it to capture dependencies between code statements and contextual logical structural features. The DCE, on the other hand, focuses on exposing crucial latent features within the code, while preserving more syntactic and semantic details. This dual-branch design strategy allows the model to simultaneously consider global structure and fine-grained features, leading to a more comprehensive understanding of smart contract code and significantly improving the accuracy and performance of vulnerability detection.

a. 

Base Transformer Encoder (BTE)

The BTE consists of multiple self-attention layers, with each self-attention layer containing multi-head self-attention and feedforward networks. It uses the multi-head self-attention feature of the Transformer encoder to capture contextual information and long-range dependencies within code sequences, thereby extracting richer global structural features. The multi-head self-attention mechanism serves as the core of BTE, allowing it to establish correlations between different positions in the input sequence and effectively capture contextual information. The structure of the BTE is shown in Figure 4.

BTE achieves parallel computation of multiple attentional heads, which allows the model to simultaneously focus on information from different positions within the input sequence. It then combines the outputs of the different heads using weighted fusion to obtain a more comprehensive feature representation. The calculation process of the BTE encoder is as follows:

$$X_{pos }=\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{a}\mathrm{l}\_\mathrm{E}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}\left(X\right)$$

(3)

$$Q=\mathrm{Linear}\left(X_{pos}\right)=X_{pos}\ast W_Q$$

(4)

$$K=\mathrm{Linear}\left(X_{pos}\right)=X_{pos}\ast W_K$$

(5)

$$V=\mathrm{Linear}\left(X_{pos}\right)=X_{pos}\ast W_V$$

(6)

$$X_{\mathrm{attention} }=\mathrm{S}\mathrm{e}\mathrm{l}\mathrm{f}\_\mathrm{Attention}\left(Q,K,V\right)$$

(7)

$X_{pos }$ is obtained using positional encoding operations, Q represents the query vector, K represents the key vector, and V represents the value vector. Each vector undergoes a linear transformation, with $W_Q$, $W_K$, and $W_V$ representing the weight matrices for these transformations. Finally, $X_{\mathrm{attention} }$ is obtained by the Equation (7).

$$X_{\mathrm{attention}1 }=X_{pos }+X_{\mathrm{attention} }$$

(8)

$$X_{\mathrm{attention}2 }= \mathrm{LayerNorm} \left(X_{\mathrm{attention}1 }\right)$$

(9)

$$X_{\mathrm{output}}=Activate\left(Linear\right(Linear\left(X_{\mathrm{attention}2}\right)\left)\right)$$

(10)

$$X_{\mathrm{output}1}=X_{\mathrm{attention}2}+X_{\mathrm{output}}$$

(11)

$$X_{\mathrm{output}2 }= \mathrm{LayerNorm} \left(X_{\mathrm{output}1}\right)$$

(12)

Then, the $X_{\mathrm{attention}2 }$ is obtained by residual connection and Layer Normalization, Finally, $X_{\mathrm{attention}2}$ processed through a feedforward neural network and residual connection to obtain $X_{\mathrm{output}2 }$ feature vectors with global structure and semantic information. This design enables the model to better understand the overall structure and global semantic relationships of smart contract code and effectively handle complex smart contracts.

b.

Detail CNN Encoder (DCE)

BTE focuses primarily on capturing global semantic information and structural features, while DCE complements BTE by focusing on extracting more intricate and complex features. DCE has the ability to delve deep into the latent syntactic, semantic and local features of the code, further enriching the feature representation of the code vectors. DCE consists of a multi-branch convolutional structure as shown in Figure 5, consisting of three branches: a 3×3 dilated convolution layer with Batch Normalization (BN), a 1×1 dilated convolution layer with BN, and an independent BN layer. Dilated convolution allows different dilation rates to be set, providing a larger receptive field without increasing the number of parameters. This helps the model to better understand the structure and relationships within the code, thus capturing more intricate details. In addition, the different kernel sizes in these convolutions allow the DCE to capture information at different scales. Finally, by fusing outputs from different branches, it integrates information from different scales and levels of detail. This helps to improve the model's perception of fine-grained details and to combine features from different hierarchical levels for a better understanding of the detailed features of the code.

In this research, we utilize symbols to represent key concepts. ${\mathrm{W}}^{\left(3\right)}\in {\mathbb{R}}^{3\times 3}$represents the convolution kernel for a 3×3 dilated convolution layer, while ${\mathrm{W}}^{\left(1\right)}\in {\mathbb{R}}^{1\times 1}$ represents the convolution kernel for a 1×1 dilated convolution layer. Additionally, we use ${\mu }^{(\cdot )}$, ${\sigma }^{(\cdot )}$, ${\gamma }^{(\cdot )}$, and ${\beta }^{(\cdot )}$ to respectively denote the accumulated mean, standard deviation, learned scaling factor, and bias for each branch in the BN layer. We have also defined symbolic representations for the input and output. ${\mathrm{X}}^{\left(1\right)}$ represents the input, and ${\mathrm{X}}^{\left(2\right)}$ represents the output. The DCE equation is as follows:

$$\begin{array}{cc}{\mathrm{X}}^{\left(2\right)}& ={\mathrm{bn}(\mathrm{X}}^{\left(1\right)}\ast {\mathrm{W}}^{\left(3\right)},{\mathsf{\mu }}^{\left(3\right)},{\mathsf{\sigma }}^{\left(3\right)},{\mathsf{\gamma }}^{\left(3\right)},{\mathsf{\beta }}^{\left(3\right)})\hfill \\ & +\mathrm{bn}({\mathrm{X}}^{\left(1\right)}\ast {\mathrm{W}}^{\left(1\right)},{\mathsf{\mu }}^{\left(1\right)},{\mathsf{\sigma }}^{\left(1\right)},{\mathsf{\gamma }}^{\left(1\right)},{\mathsf{\beta }}^{\left(1\right)})\hfill \\ & +\mathrm{b}\mathrm{n}({\mathrm{X}}^{\left(1\right)},{\mathsf{\mu }}^{\left(0\right)},{\mathsf{\sigma }}^{\left(0\right)},{\mathsf{\gamma }}^{\left(0\right)},{\mathsf{\beta }}^{\left(0\right)})\hfill \end{array}$$

(13)

In summary, the use of DCE enhances the model's ability to perceive and capture these intricate and complex features, thereby helping the model to better distinguish between vulnerable code and normal code.

4.3.3. Deep Residual Shrinkage Network

The DRSN module integrates soft thresholding and deep learning, introducing a method for adaptive thresholding [31]. Through DRSN, the model can effectively reduce irrelevant information during feature learning. The calculation of the soft threshold is as follows:

(14)

The soft thresholding calculation is shown in Figure 6b, where $x$ represents the input features, $y$ represents the output features, and $\tau$ is the threshold, which is a positive parameter. As can be seen from the Equation (14), the essence of soft thresholding is to discard features with smaller absolute values and shrink features with larger absolute values, thereby reducing information unrelated to the current task.

The structure of DRSN is shown in Figure 6c. First, the input vector passes two convolution layers, each accompanied by normalization layers and Relu activation functions. If the output of the convolution layer is a matrix is denoted as $X\in R^{T\times K}$, where T is the number of token sets, and K is the number of convolution kernels. The resulting matrix undergoes an absolute value operation, followed by global average pooling to generate a one-dimensional vector, denoted as $X_{avg}\in R^K$.

$$X_{avg}=\frac{1}{T}{\sum }_{i=1}^T\mid X_i\mid$$

(15)

Where $\mid X_i\mid$denotes the absolute value of the${ i}^{th}$ row feature in $X$. Subsequently, $X_{avg}$ is fed through a fully connected layer, followed by batch normalization, a Relu activation function, and a second fully connected layer. Further, a sigmoid activation function is used to scale the threshold parameter for each channel in the range (0, 1), resulting in the scaling parameter $sc\in R^K$, and the threshold $\tau \in R^K$ can be calculated using Equation (17).

$$sc=\frac{1}{1+e^{-Z}}$$

(16)

$$\tau =sc\ast X_{avg}$$

(17)

After processing by DRSN, which eliminates residual redundant information and ensures that the feature vectors retain only crucial information related to the vulnerability characteristics. This helps to improve the performance of the model, enabling it to better distinguish between vulnerable and non-vulnerable samples, thereby increasing the accuracy and effectiveness of vulnerability detection.

5. Experiments

6. Conclusions

References

1. Alharby, M.; Moorsel, A.V. Blockchain Based Smart Contracts: A Systematic Mapping Study. In Proceedings of the Computer Science & Information Technology (CS & IT); Academy & Industry Research Collaboration Center (AIRCC), August 26 2017; pp. 125–140. [Google Scholar] [CrossRef]
2. Gupta, R.; Tanwar, S.; Al-Turjman, F.; Italiya, P.; Nauman, A.; Kim, S.W. Smart Contract Privacy Protection Using AI in Cyber-Physical Systems: Tools, Techniques and Challenges. IEEE Access 2020, 8, 24746–24772. [Google Scholar] [CrossRef]
3. Wang, S.; Ouyang, L.; Yuan, Y.; Ni, X.; Han, X.; Wang, F.-Y. Blockchain-Enabled Smart Contracts: Architecture, Applications, and Future Trends. IEEE Trans. Syst. Man Cybern, Syst. 2019, 49, 2266–2277. [Google Scholar] [CrossRef]
4. Khan, S.N.; Loukil, F.; Ghedira-Guegan, C.; Benkhelifa, E.; Bani-Hani, A. Blockchain Smart Contracts: Applications, Challenges, and Future Trends. Peer-to-Peer Netw. Appl. 2021, 14, 2901–2925. [Google Scholar] [CrossRef] [PubMed]
5. Wu, H.; Dong, H.; He, Y.; Duan, Q. Smart Contract Vulnerability Detection Based on Hybrid Attention Mechanism Model. Applied Sciences 2023, 13, 770. [Google Scholar] [CrossRef]
6. Mehar, M.I.; Shier, C.L.; Giambattista, A.; Gong, E.; Fletcher, G.; Sanayhie, R.; Kim, H.M.; Laskowski, M. Understanding a Revolutionary and Flawed Grand Experiment in Blockchain: The DAO Attack. Journal of Cases on Information Technology (JCIT) 2019, 21, 19–32. [Google Scholar] [CrossRef]
7. Almakhour, M.; Sliman, L.; Samhat, A.E.; Mellouk, A. Verification of Smart Contracts: A Survey. Pervasive and Mobile Computing 2020, 67, 101227. [Google Scholar] [CrossRef]
8. He, D.; Deng, Z.; Zhang, Y.; Chan, S.; Cheng, Y.; Guizani, N. Smart Contract Vulnerability Analysis and Security Audit. IEEE Network 2020, 34, 276–282. [Google Scholar] [CrossRef]
9. Chu, H.; Zhang, P.; Dong, H.; Xiao, Y.; Ji, S.; Li, W. A Survey on Smart Contract Vulnerabilities: Data Sources, Detection and Repair. Information and Software Technology 2023, 159, 107221. [Google Scholar] [CrossRef]
10. Qian, P.; Liu, Z.; He, Q.; Zimmermann, R.; Wang, X. Towards Automated Reentrancy Detection for Smart Contracts Based on Sequential Models. IEEE Access 2020, 8, 19685–19695. [Google Scholar] [CrossRef]
11. Cai, J.; Li, B.; Zhang, J.; Sun, X.; Chen, B. Combine Sliced Joint Graph with Graph Neural Networks for Smart Contract Vulnerability Detection. Journal of Systems and Software 2023, 195, 111550. [Google Scholar] [CrossRef]
12. Zhang, L.; Chen, W.; Wang, W.; Jin, Z.; Zhao, C.; Cai, Z.; Chen, H. CBGRU: A Detection Method of Smart Contract Vulnerability Based on a Hybrid Model. Sensors 2022, 22, 3577. [Google Scholar] [CrossRef] [PubMed]
13. Wu, H.; Zhang, Z.; Wang, S.; Lei, Y.; Lin, B.; Qin, Y.; Zhang, H.; Mao, X. Peculiar: Smart Contract Vulnerability Detection Based on Crucial Data Flow Graph and Pre-Training Techniques. In Proceedings of the 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE); IEEE: Wuhan, China, October 2021; pp. 378–389. [Google Scholar] [CrossRef]
14. Li, M.; Ren, X.; Fu, H.; Li, Z.; Sun, J. ConvMHSA-SCVD: Enhancing Smart Contract Vulnerability Detection through a Knowledge-Driven and Data-Driven Framework. In Proceedings of the 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE); IEEE: Florence, Italy, 9 October 2023; pp. 578–589. [Google Scholar] [CrossRef]
15. Yu, L.; Lu, J.; Liu, X.; Yang, L.; Zhang, F.; Ma, J. PSCVFinder: A Prompt-Tuning Based Framework for Smart Contract Vulnerability Detection. In Proceedings of the 2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE); IEEE: Florence, Italy, 9 October 2023; pp. 556–567. [Google Scholar] [CrossRef]
16. Atzei, N.; Bartoletti, M.; Cimoli, T. A Survey of Attacks on Ethereum Smart Contracts (Sok). In Proceedings of the Principles of Security and Trust: 6th International Conference, POST 2017, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2017, Uppsala, Sweden, April 22-29, 2017, Proceedings 6; Springer, 2017; pp. 164–186. [CrossRef]
17. Liu, Z.; Qian, P.; Wang, X.; Zhuang, Y.; Qiu, L.; Wang, X. Combining Graph Neural Networks with Expert Knowledge for Smart Contract Vulnerability Detection. IEEE Trans. Knowl. Data Eng. 2021, 1–1. [Google Scholar] [CrossRef]
18. Jiang, B.; Liu, Y.; Chan, W.K. Contractfuzzer: Fuzzing Smart Contracts for Vulnerability Detection. In Proceedings of the Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering; 2018; pp. 259–269. [CrossRef]
19. Tsankov, P.; Dan, A.; Drachsler-Cohen, D.; Gervais, A.; Buenzli, F.; Vechev, M. Securify: Practical Security Analysis of Smart Contracts. In Proceedings of the Proceedings of the 2018 ACM SIGSAC conference on computer and communications security; 2018; pp. 67–82. [CrossRef]
20. Luu, L.; Chu, D.-H.; Olickel, H.; Saxena, P.; Hobor, A. Making Smart Contracts Smarter. In Proceedings of the Proceedings of the 2016 ACM SIGSAC conference on computer and communications security; 2016; pp. 254–269. [CrossRef]
21. Prechtel, D.; Groß, T.; Müller, T. Evaluating Spread of ‘Gasless Send’in Ethereum Smart Contracts. In Proceedings of the 2019 10th IFIP international conference on new technologies, mobility and security (NTMS); IEEE, 2019; pp. 1–6. [CrossRef]
22. Feist, J.; Grieco, G.; Groce, A. Slither: A Static Analysis Framework for Smart Contracts. In Proceedings of the 2019 IEEE/ACM 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB); IEEE, 2019; pp. 8–15. [CrossRef]
23. Yu, X.; Zhao, H.; Hou, B.; Ying, Z.; Wu, B. DeeSCVHunter: A Deep Learning-Based Framework for Smart Contract Vulnerability Detection. In Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN); IEEE: Shenzhen, China, July 18 2021; pp. 1–8. [CrossRef]
24. Liu, Z.; Qian, P.; Wang, X.; Zhu, L.; He, Q.; Ji, S. Smart Contract Vulnerability Detection: From Pure Neural Network to Interpretable Graph Feature and Expert Pattern Fusion. arXiv 2021, arXiv:2106.09282. [Google Scholar] [CrossRef]
25. Duy, P.T.; Khoa, N.H.; Quyen, N.H.; Trinh, L.C.; Kien, V.T.; Hoang, T.M.; Pham, V.-H. VulnSense: Efficient Vulnerability Detection in Ethereum Smart Contracts by Multimodal Learning with Graph Neural Network and Language Model. arXiv 2023, arXiv:2309.08474. [Google Scholar]
26. Sak, H.; Senior, A.W.; Beaufays, F. Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling. 2014.
27. Kipf, T.N.; Welling, M. Semi-Supervised Classification with Graph Convolutional Networks. arXiv 2016, arXiv:1609.02907. [Google Scholar]
28. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, \Lukasz; Polosukhin, I. Attention Is All You Need. Advances in neural information processing systems 2017, 30.
29. Zamir, S.W.; Arora, A.; Khan, S.; Hayat, M.; Khan, F.S.; Yang, M.-H. Restormer: Efficient Transformer for High-Resolution Image Restoration. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); IEEE: New Orleans, LA, USA, June 2022; pp. 5718–5729.
30. Ding, X.; Zhang, X.; Ma, N.; Han, J.; Ding, G.; Sun, J. RepVGG: Making VGG-Style ConvNets Great Again. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); IEEE: Nashville, TN, USA, June 2021; pp. 13728–13737.
31. Zhao, M.; Zhong, S.; Fu, X.; Tang, B.; Pecht, M. Deep Residual Shrinkage Networks for Fault Diagnosis. IEEE Transactions on Industrial Informatics 2019, 16, 4681–4690. [Google Scholar] [CrossRef]
32. Zhao, Z.; Bai, H.; Zhang, J.; Zhang, Y.; Xu, S.; Lin, Z.; Timofte, R.; Van Gool, L. CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion. In Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); IEEE: Vancouver, BC, Canada, June 2023; pp. 5906–5916.
33. Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G.S.; Dean, J. Distributed Representations of Words and Phrases and Their Compositionality. Advances in neural information processing systems 2013, 26. [Google Scholar]
34. Zhang, L.; Li, Y.; Jin, T.; Wang, W.; Jin, Z.; Zhao, C.; Cai, Z.; Chen, H. SPCBIG-EC: A Robust Serial Hybrid Model for Smart Contract Vulnerability Detection. Sensors 2022, 22, 4621. [Google Scholar] [CrossRef] [PubMed]
35. Durieux, T.; Ferreira, J.F.; Abreu, R.; Cruz, P. Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts. In Proceedings of the Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering; June 27 2020; pp. 530–541. [CrossRef]
36. Zhuang, Y.; Liu, Z.; Qian, P.; Liu, Q.; Wang, X.; He, Q. Smart Contract Vulnerability Detection Using Graph Neural Network. In Proceedings of the Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence; International Joint Conferences on Artificial Intelligence Organization: Yokohama, Japan, July 2020; pp. 3283–3290.