We designed a multi-behavior attention network to help users collaborate on different behaviors. In order to make the behavior interaction context as the input of the sequence path, the embedding layer of the behavior context sequence path is designed. Multi Behavioral Interaction Sequence (MB Sequence) is used as the input to the sequence path, and we design the embedding layer for the behavioral contextual sequence path. Joint embedding of individual user, project information and corresponding interactive behavior context information. Given $u_q\in U$, we set each element in the sequence of user interaction as a triple $s_{u_q,i_x}^{b_k}=\left\{(u_q,b_k,i_x)|u_q\in U,b_k\in B,i_x\in I\right\}$, which indicates that user q interacts with item x with the k-th behavior. The multi-behavior sequence of a user contains the interaction information of a single user. Thus, the multi-behavior interaction sequence of the user is mapped to the initial embedding matrix $S_{u_q}=[\left(u_q,b_1,i_x\right),\left(u_q,b_2,i_x\right),\dots ,\left(u_q,b_K,i_x\right)]$ as shown in Figure 2.

In order to better extract user preferences and simulate the dependence between different types of behaviors, we use attention in user behavior interaction to obtain the commonality of individual behaviors and the relationship of different behaviors. Considering the correlation between different behaviors of users, this correlation will help e-commerce platforms to combine the correlation of behaviors to more accurately recommend products to users, and the attention can capture which behavior has a greater relationship with the purchase behavior, hence the name "multi-behavior attention". The relationship between multiple user behaviors is used to model the fine-grained interaction between users and items, as shown in Figure 2. We propose a Multi-Behavior Interaction Sequence-Dependent Encoder(MB Dependency Encoder), which is an implicit or explicit user interest about items that users interact with by using within behaviors and between behaviors. Different behaviors of users differently contribute to items that users may be interested in. Therefore, we propose to capture the different modal relationships using user behaviors in attention. Based on self-attention mechanism capture fine-grained correlations between different types of auxiliary behaviors and target behaviors, modeling multi-behavior dependencies. Multiple behaviors are put into attention can make behaviors no longer independent behaviors, and form a vector of features related to the purchase behavior. Specifically, $S_{u_q}$ is a sequence of historical interactions of each user. The input of attention is an auxiliary behavior feature vector $s_{u_q,i_1}^{b_k}\in R^{n*d}$ and the target behavior feature vector $s_{u_q,i_1}^{b_{k^{\prime }}}$ of the same user, $k\in [1,2,\dots ,K]$, where $i_1$ represents the first item i in the item collection I. The feature vector $s_{u_q,i_1}^{b_k}$ is an element in the sequence $S_{u_q}$. In addition, we calculate the weight matrix of each auxiliary behavior feature vector $s_{u_q,i_1}^{b_k}$ and target behavior feature vector $s_{u_q,i_1}^{b_{k^{\prime }}}$, which is to understand the correlation between behaviors from users’ historical behaviors: where $W^Q$,$W^K$,$W^V$$\in R^{d*n}$ are weight matrices of learnable behavior vectors. As $A_{k,k^{\prime }}^s\in R^{n*n}$ represents an correlation matrix between the auxiliary behavior k and the target behavior $k^{\prime }$. This correlation matrix represents the correlation between the auxiliary and target behaviors. ${\alpha }_k^s$ represents the attention score. $A_{k,k^{\prime }}^s$ represents calculated by $softmax$ function in order to find auxiliary behavior that is closer to the target behavior. To prevent overfitting problems, we use dropout method to obtain ${\alpha }_k^s$. The sequence $s_q^{b_{k,k^{\prime }}}$ contain the closer relationship between the auxiliary behavior and the target behavior ,which is obtained by multiplying the attention fraction ${\alpha }_k^s$ with the behavior feature vector $s_{u_q,i}^{b_k}$.

The history of multiple user interactions in the real world contains a wealth of information. Past studies have modeled user interactions chronologically to predict the next item that a user will interact with. However, due to the existence of many external uncertain factors, it is impossible for users to interact according to the strict sequence of interactive behaviors. We developed a multi-behavior Attention Fusion to extract different user behavior characteristics, which is to capture personalized user interests by modeling the underlying relationships of different types of behavior. We develop an Attention Fusion to extract different user behavior features. Inspired by BERT4Rec [7] in terms of text, we introduce a semantic encoder of behavior to capture personalized user interests: where we use $Gaussianerrorlinearelement$ (GELU) activation function. W represents the weight matrix of the GELU activation $function$, b means bias. $s_q^{b_{k,k^{\prime }}}$ represents the user partial long-term interest captured by k behaviors in the user long-term history. Softmax, as the activation function of output, normalizes the results of the concatenation of various behavior sequences:

For different users, users have different behavioral interaction sequences leading to different encoding results.

In TP_MB, it is our goal to captures the commonality of multiple behaviors of the same user, the differences of multiple behaviors of different users. Inspired by EHCF, users’ multi-behavioral relationships reflect their implicit intentions[13]. Self-supervised learning is a kind of contrast learning in machine learning. Therefore, we design intra-path self-supervised learning of multi-behavior interaction sequences to enhance supervision signals of multiple behavioral data through self-supervised learning. Specifically, we regard the different behaviors of the same user as positive sample pairs$\left\{\left(s_q^{b_k},s_q^{b_{k+1}}\right)|q\in U\right\}$, and the different behaviors of different users as negative sample pairs$\left\{\left(s_q^{b_k},s_p^{b_{k+1}}\right)|q,p\in U,q\ne p\right\}$. We define the TP_MB objective $function$ according to InfoNCE:

In general, using a single information of the user cannot obtain enough available information. Since users have multiple behaviors, including clicking, shopping cart, favorate and purchase. In addition, for the user’s purchase behavior, other behaviors have great value in helping extract user interest. Therefore, we set the purchase behavior as the target behavior, and other behaviors as auxiliary behaviors. Learning user interests with the help of auxiliary behaviors is to predict user purchasing behavior. Here we build a user-item graph of different interaction behaviors through all items that users interact with in different behaviors. Multi-behavior interaction graph(MB Interaction Graph) contain all information about user interactions. First, we define $G=(V,E)$, V means that the node set includes user set $u\in U$ and item set $i\in I$, namely $(U,I)\in V$. E represents different interaction behaviors between user nodes and item nodes. Furthermore, the embedding of a multi-behavior graph is composed of multiple behavior subgraph embeddings, so our behavior subgraph embedding can be expressed as $G_b=(V_b,E_b)$. For example, the behavior subgraph composed of the items clicked by the user, $G_{click}=\left(V_{click},E_{click}\right)$,$G_{click}$ represents the item graph representation that the user interacts with by clicking. $V_{click}$ represents the user and item node connected with the click behavior, and $E_{click}$ represents the user’s click behavior.

We design Multi-Behavior Interaction Graph Information Aggregation. Research results on graph convolution showed that general research focused on node characterization of simple undirected graphs[8]. The above work ignored edges between nodes that also had multiple properties, such as behavior. Inspired by graph neural network processing user multi-relational data GHCF [2]. We propose a multi-behavior interaction graph path, and use neural networks to obtain the global representation of user interaction, as shown in Figure 3. Specifically, it first focuses on the node representation of the learning graph. For each behavior subgraph, it is embedded into the adjacency matrix $A_k$, which is composed of the matrix $R\_k$: where $D_k^{-\frac{1}{2}}$ represents the degree matrix of k behavior, and $I_k$ represents the identity matrix of k behavior. $L_k$ is the normalized Laplacian matrix of k behavior. For the multi-behavior interaction graph of users, GCN can be used to better obtain the global representation of all users. We follow LightGCN: where $x_{b_k}^{\left(l\right)}$ is the node feature matrix of layer l of nodes in the figure, and $W_k$ is the transformation matrix of behavior graph information transfer. $Sigmoid$ is the activation $function$. GCN has a total of L layers, and l represents the order of neighbors we obtain. By obtaining the features of nodes with k behaviors in the graph. We design a multi-behavior interaction sequence attention encoder. The encoder constructs with the graph node representation under different behaviors. The influence coefficient of the corresponding auxiliary behavior graph with respect to the target behavior graph can be obtained. Therefore, we introduce a cross-behavior interaction graph attention encoder, auxiliary behavior graph $x_{b_k}$ and target behavior graph $x_{b_{k^{\prime }}}$ as input: where $W^Q\in R^{d*n}$,$W^K\in R^{d*n}$,$W^V\in R^{d*n}$ is the weight matrix of the learnable behavior matrix. ${\alpha }_k^g$ g is the attention correlation coefficient, which is regarded as the weight added to the auxiliary behavior $x_{b_k}$. $A_{k,k^{\prime }}^g$ is calculated in the same way as we mentioned in section 3.2. By taking the Attention-Correlation matrix $A_{k,k^{\prime }}^g$ as the input, it is obtained by Multi-Classification. $X_{b_k}$ is the auxiliary behavior feature matrix about the target behavior as the final output of the cross-behavior interaction graph attention encoder.

As we all known, the sparsity problem of user interaction behavior in the recommendation system was widespread[15]. There is also a long tail problem among behaviors. For example, users click too many times and make few purchases. Therefore, to alleviate this problem, in TP_MB we capture user interests by capturing complex dependencies among behaviors through unsupervised signals of multi-behavior interaction graphs, instead of utilizing a single behavior to capture user interests. Specifically, the output of our multi-behavior interaction graph attention encoder and the embedding of the graph of the target behavior are taken as input. That is to say, the embedding $X_k$ of the graph of the auxiliary behavior and the embedding $X_{k^{\prime }}$ of the graph of the target behavior are used as input. Contrastive learning is one of the method in self-supervised learning. Thus, features are learned by constructing positive and negative examples and establishing a contrastive loss. Therefore, intra-path multi-behavior interaction graph self-supervised learning is designed to enhance multiple behavioral data supervision signals through self-supervised learning. Graph self-supervised learning based on InfoNCE can effectively alleviate the long-tail distribution problem of different behaviors among users. In TP_MB, different behavior graph of the same user are regarded as positive sample pairs$\left\{\left(X_q^k,X_q^{k+1}\right)|q\in U\right\}$, and different behaviors of different users are regarded as negative sample pairs$\left\{\left(X_q^k,X_p^{k+1}\right)|q,p\in U,q\ne p\right\}$. Maximize mutual information between users by comparing positive and negative sample pairs:

The consistency of two behavioral graph is maximized via the self-supervised loss defined above. And maximize the difference between different user behaviors to get behavior data enhancement.