1. Introduction

2. Related Work

3. Problem Statement

4. Proposed Method

4.3. Prediction Model

Our Prediction model architecture is shown in Figure 7, it uses the features mentioned in input features to obtain the predicted next location. In order to better learn temporal features and location features, we use Time2vec to learn the periodic features of the temporal features and use Node2vec to extract contextual features rich in trajectory sequences. Next, the temporal features, location features, and semantic features are concatenated as input for the prediction model. In order to predict the next location, we adopt Self-Attention to solve this issue. Self-Attention is a model that has been successfully used for time series forecasting challenges and can model temporal properties well. In order to personalize our model, we add the user ID features to the output of Self-Attention. Finally, we can obtain the final prediction result after softmax.

4.3.1. Temporal Features Extractor

The temporal feature t is a one-dimensional vector represented by hours (1-24). However, time is a periodic feature. Although using hours to represent the temporal features can reflect the periodicity, but it may not be able to fully express the period of time. Therefore, inspired by Kazemi et al. [9], we adopt Time2vec to extract the periodic temporal features. The formula of Time2vec is shown as (3), where $\mathbb{t}$ is the dimension of the time2vec output $B_t^{*}$. The first $b_{t_1}(i=0)$ dimension learns original temporal features, the other dimensions $b_{t_2},\dots ,b_{t_{\mathbb{t}}}(1\le i\le \mathbb{t})$ learns periodic temporal features using the sine function. Where ${\omega }_i$and ${\phi }_i$ are learnable parameters, referring to the phase-shift and frequency of the sine function.

$$\begin{gathered} b_{t_i}=Time2vec\left(t\right)\left[i\right]=\left\{\begin{array}{c}{\omega }_i\tau +{\phi }_i, if i=0\\ \sin \left({\omega }_i\tau -{\phi }_i\right), if 1\le i\le t\end{array}\right. \\ {B}_t^{*}=\left[b_{t_1},b_{t_2},\dots ,b_{t_{\mathbb{t}}}\right], B_t^{*}\in {\mathbb{R}}^{{d_{\mathbb{t}}}_{ }} \end{gathered}$$

(3)

4.3.2. Location Features Extractor

The location feature $\mathrm{g}$ is a one hot vector (the dimension is composed of the number of grids covering the study area). Due to the large number of grids, the dimension of $\mathrm{g}$ is very huge, which may lead to the curse of dimensionality, not only increasing the amount of computation, but also making the model easier to overfit. Therefore, we adopt a graph-based embedding method called Node2vec, which encodes each grid in all trajectories to construct a directed weighted graph. It then takes the number of visits to the location as the weight of the graph, which reflects the visit order preference and frequency; while using Breadth-First Search (BFS), Depth-First Search (DFS), and random walks to generate training data. Finally, using Word2vec embeddings to obtain more expressive features that fully consider the interaction between each location and its neighbors, it also can capture richer context spatial information.

In Node2vec, for all trajectories of all users, we create a directed weighted $Graph=(V,E, W)$. Where$V$is a set of vertices (all grids will be a vertices), and $E$ is a set of edges. We define edge $e_{ij}$ as (4).

$$e_{ij}=\left\{\begin{array}{c}1, from{ g}_i to g_j\\ 0, unachievable\end{array}\right.{ ,e}_{ij}\in E$$

(4)

Based on the trajectory movement of all users, we can get $E$. The $w_{ij}\in W$ denotes the weight of $e_{ij}$, which is calculated from the total number of visits. Then, Node2vec combines the BFS and DFS to sample the graph. Given the current node $v \in V$, the transition probability of visiting the next node $x$ is shown as (5).

$$P\left(c_i=x \right| c_{i-1}=v)=\left\{\begin{array}{c}\frac{{\pi }_{vx}}{Z}, if\left(v,x\right) ϵ E\\ 0, otherwise\end{array}\right.$$

(5)

Before calculating the transition probability, node $v$ and node $x$ must have an edge link; ${\pi }_{vx}$is the probability of an unnormalized transition between node $v$ and node $x$, whereas $Z$ refers to the normalization constant. In order to control the random walk strategy, Node2vec uses two hyperparameters (p and q), which is defined as (6).

$$\begin{gathered} {\pi }_{vx}={\alpha }_{pq}\left(\tilde{t},x\right)·w_{vx} \\ {\alpha }_{pq}\left(\tilde{t},x\right)=\left\{\begin{array}{c}\frac{1}{p},if{ d}_{\tilde{t}x}=0 \\ 1,if{ d}_{\tilde{t}x}=1\\ \frac{1}{q},if{ d}_{\tilde{t}x}=2\end{array}\right. \end{gathered}$$

(6)

The random walk strategy is demonstrated in Figure 8. Where $w_{vx}$ is the weight of edge $e_{vx}$, and $d_{\tilde{t}x}$ is the shortest number of steps between the previous node $\tilde{t}$ and the next node $x$. When node $v$ transfers to node $x$, we have to consider the number of steps between the previous node $\tilde{t}$ and next node $x$.

The parameter $p$ determines the probability that $v$ will visit the previous node $\tilde{t}$, and $p$ only functions when $d_{\tilde{t}x}$= 0. The probability of accessing the previous node $\tilde{t}$ is larger if $p$ has a large value. The random walk's direction is determined by the parameter $q$; $q>1$ causes the random walk to tend nodes near node $\tilde{t}$ (BFS); $q$ only functions when $d_{\tilde{t}x}$= 2. The random walk has a propensity to travel to nodes far from node $\tilde{t}$ (DFS) if $q<1$. The loss of Node2vec is calculated as (7), where $N_s\left(\tilde{u}\right)$ is the set of neighboring nodes of node $\tilde{u}$ obtained by sampling strategy and $f\left(\tilde{u}\right)$ is a mapping function that maps node $\tilde{u}$ to an embedding vector. For the current node $f\left(\tilde{u}\right)$, the optimization goal is to give each node $f\left(\tilde{u}\right)$ the condition to maximize the probability of the adjacent node $N_s\left(\tilde{u}\right)$.

$$L_{Node2vec}={max}_f\sum _{u\in V}log\mathrm{Pr}\left( N_s\left(\tilde{u}\right) \right| f\left(\tilde{u})\right)$$

(7)

Then according to this loss $L_{Node2vec}$, the skip-gram in word2vec is directly used to learn the embedding vector. The principle of Word2vec is to use the weights of the fully connected layer of a classifier as word vectors. In this classifier, the word’s one-hot vectors are input, then fed into a fully connected layer, and then fed to a softmax to get the probability of the contexts. We followed previous work [16] and set $p=q=0.25$. The obtained location embedding vector is defined as (8), where $\mathbb{g}$ is the dimension of Node2vec output that can be determined, and $B_g$is the location feature extracted by Node2vec. We use Node2vec to obtain more expressive location features which fully considers the relationship between each node and its neighbor nodes.

$$Node2vec\left(V,E\right)=B_g^{*}=\left[b_{g_1},b_{g_2},\dots ,b_{g_{\mathbb{g}}}\right], B_g^{*}\in {\mathbb{R}}^{d_{\mathbb{g}}}$$

(8)

4.3.3. Model Structure

We fed the features into the model to obtain the next location. The output of Time2vec is $B_t\in {\mathbb{R}}^{d_h\times d_{\mathbb{t}}}$, and the output of Node2vec $B_g\in {\mathbb{R}}^{d_h\times d_{\mathbb{g}}}$. The semantic $\varsigma$ and user id $u$ are fed into the fully connected layer to obtain $B_{\varsigma }\in {\mathbb{R}}^{d_h\times d_{\mathbb{s}}}$and $B_u\in {\mathbb{R}}^{d_h\times d_{\mathbb{u}}}$ as (9), where $W^{\varsigma }\in {\mathbb{R}}^{d_{\varsigma }\times d_{\mathbb{s}}}$ and $W^u\in {\mathbb{R}}^{d_u\times d_{\mathbb{u}}}$ are trainable weight matrices, $b_{\varsigma }$ and $b_u$ are the bias parameter of the fully connected layer. Our prediction model input $\mathit{X}=\left\{{\mathit{X}}_1,{\mathit{X}}_2,\dots ,{\mathit{X}}_h\right\}$ is shown as (10).

(9)

(10)

$B_u$ is not directly trained by the prediction model because $B_u$ does not have a time series characteristic; how $B_u$ is considered by the model will be described later. In order to capture long-term dependencies in trajectory sequences, we adopt the Self-Attention model. Self-Attention use fully connected layer to consider the information from each step collectively, which allows it to maintain more long-term memory than RNN, which communicates information at each stage. Self-Attention also has fewer parameters and a lower time complexity when compared to RNN. The formula for Self-Attention is shown as (11), where $W^{\mathbb{Q}}\in {\mathbb{R}}^{d\times d_{\mathbb{q}}}, W^{\mathbb{K}}\in {\mathbb{R}}^{d\times d_{\mathbb{k}}} ,W^{\mathbb{V}}\in {\mathbb{R}}^{d\times d_{\mathbb{v}}}$are weight matrices that can be trained. $\mathit{X}$ is fed to 3 full connections to get $\mathbb{Q}\in {\mathbb{R}}^{d_h\times d_{\mathbb{q}}},\mathbb{K}\in {\mathbb{R}}^{d_h\times d_{\mathbb{k}}},\mathbb{V}\in {\mathbb{R}}^{d_h\times d_{\mathbb{v}}}$. $d_{\mathbb{x}}$ ($d_{\mathbb{q}}$ = $d_{\mathbb{k}}$ = $d_{\mathbb{v}}=$ $d_{\mathbb{x}}$) is the dimension of the hidden state of Self-Attention. We then compute the dot product of $\mathbb{Q}$ with ${\mathbb{K}}^T$(calculate the features similarity of each location in a trajectory sequence), divide by $\sqrt{d_{\mathbb{k}}}$ (the purpose of dividing by $\sqrt{d_{\mathbb{k}}}$ is to make the gradient of softmax not too small), apply a softmax function (the features similarity of each stage is expressed by probability, and the higher the features similarity is, the more important the features of the stage is), then dot product $\mathbb{V}$ (get a weighted score for each location, this determines which stage features are important) to obtain ${\mathit{X}}^{\mathit{*}}\in {\mathbb{R}}^{d_h\times d_{\mathbb{x}}}$.

(11)

Finally, we only take the last stage ${\mathit{X}}_h^{\mathit{*}}\in$ ${\mathit{X}}^{\mathit{*}}$ as the output of Self-Attention, so the dimension of ${\mathit{X}}_h^{\mathit{*}}$ is $1\times d_{\mathbb{x}}$. There are two reasons we only take the last stage ${\mathit{X}}_h^{\mathit{*}}$ : Each stage uses the fully connection to consider all spatiotemporal contexts of the sequence; if the output is $h$ stages, the number of network parameters will increase, which may easily cause the problem of overfitting.

In order to personalize the prediction model, we consider the user ID in the output of Self-Attention, and then feed it to softmax to get the final output. The formula is shown as (12), where $W^o\in {\mathbb{R}}^{d_{\mathbb{x}}\times d_{\mathbb{o}}},W^y\in {\mathbb{R}}^{1\times d_g}$ are the weight matrices, $b_o$ and $b_y$ are the bias parameters of the fully connection. We feed ${\mathit{X}}_h^{\mathit{*}}$ to the fully connection to get the output vector $O\in {\mathbb{R}}^{1\times d_{\mathbb{o}}}$. The trajectory patterns of each user are different, and the model may mix them together during training. In order to distinguish the trajectory sequence of each user, $O$ is added to the feature of the user ID $B_u$ to obtain $O_u$, so that the model can distinguish the trajectory pattern of each user. Finally, $O_u$ is fed to the fully connection, so that the dimension of the output is the same as the number of grids $g$. After feeding to softmax, the final output of the model $y\in {\mathbb{R}}^{1\times d_g}$ can be obtained.

(12)

5. Experimental Evaluation

6. Conclusions and Future Work

References

1. Al-Molegi, A.; Jabreel, M.; Martínez-Ballesté, A. Move, Attend and Predict: An Attention-Based Neural Model for People’s Movement Prediction. Pattern Recognition Letters 2018, 112, 34–40. [Google Scholar] [CrossRef]
2. Chen, C. C.; Chiang, M. F. Trajectory Pattern Mining: Exploring Semantic and Time Information. In Proceedings of the IEEE Conference on Technologies and Applications of Artificial Intelligence, Hsinchu, Taiwan, Nov. 25-27 2016; pp. 130–137. [Google Scholar]
3. Chen, M.; Zuo, Y.; Jia, X.; Liu, Y.; Yu, X.; Zheng, K. CEM: A Convolutional Embedding Model for Predicting Next Locations. IEEE Transactions on Intelligent Transportation Systems 2020, 23, 3349–3358. [Google Scholar] [CrossRef]
4. Cao, H., Xu; F., Sankaranarayanan, J.; Li, Y.; Samet, H. Habit2vec: Trajectory Semantic Embedding for Living Pattern Recognition in Population. IEEE Transactions on Mobile Computing 2019, 19, 1096–1108.
5. Fernandes, R.; GL, R. D. S. A New Approach to Predict User Mobility Using Semantic Analysis and Machine Learning. Journal of Medical Systems 2017, 41, 1–12. [Google Scholar] [CrossRef] [PubMed]
6. Feng, J.; Li, Y.; Yang, Z.; Qiu, Q.; Jin, D. Predicting Human Mobility with Semantic Motivation via Multi-Task Attentional Recurrent Networks. IEEE Transactions on Knowledge and Data Engineering 2022, 34, 2360–2374. [Google Scholar] [CrossRef]
7. Grover, A.; Leskovec, J. Node2vec: Scalable Feature Learning for Networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, California, USA, Aug. 13-17 2016; pp. 855–864. [Google Scholar]
8. Jiang, J.; Pan, C.; Liu, H.; Yang, G. Predicting Human Mobility Based on Location Data Modeled by Markov Chains. In Proceedings of the IEEE Fourth International Conference on Ubiquitous Positioning, Indoor Navigation and Location Based Services, Shanghai, China, Nov. 3-4 2016; pp. 145–151. [Google Scholar]
9. Kazemi, S. M.; Goel, R.; Eghbali, S.; Ramanan, J.; Sahota, J.; Thakur, S.; Wu, S.; Smyth, C.; Poupart, P.; Brubaker, M. Time2vec: Learning A Vector Representation of Time. arXiv preprint arXiv:1907.05321, 2019.
10. Lu, E. H. C.; Lin, Z. Q. Rental Prediction in Bicycle-Sharing System Using Recurrent Neural Network. IEEE Access 2020, 8, 92262–92274. [Google Scholar] [CrossRef]
11. Mikolov, T.; Chen, K.; Corrado, G.; Dean, J. Efficient Estimation of Word Representations in Vector Space. arXiv preprint arXiv:1301.3781, 2013.
12. Sassi, A.; Brahimi, M.; Bechkit, W.; Bachir, A. Location Embedding and Deep Convolutional Neural Networks for Next Location Prediction. In Proceedings of the IEEE 44th LCN Symposium on Emerging Topics in Networking, Osnabrück, Germany, Oct. 14-17 2019; pp. 149–157. [Google Scholar]
13. Su, L.; Li, L. Trajectory Prediction Based on Machine Learning. IOP Conference Series: Materials Science and Engineering 2020, 790, 012032. [Google Scholar] [CrossRef]
14. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; Polosukhin, I. Attention Is All You Need. In Proceedings of the Advances in Neural Information Processing Systems, Long Beach, California, USA, Dec. 4-9 2017; pp. 5998–6008. [Google Scholar]
15. Wang, S.; Li, A.; Xie, S.; Li, W.; Wang, B.; Yao, S.; Asif, M. A Spatial-Temporal Self-Attention Network (STSAN) for Location Prediction. Complexity 2021, 6692313. [Google Scholar] [CrossRef]
16. Wen, S.; Zhang, X.; Cao, R.; Li, B.; Li, Y. MSSRM: A Multi-Embedding Based Self-Attention Spatio-Temporal Recurrent Model for Human Mobility Prediction. Human-Centric Computing and Information Sciences 2021, 11. [Google Scholar] [CrossRef]
17. Xia, L.; Huang, Q.; Wu, D. Decision Tree-based Contextual Location Prediction from Mobile Device Logs. Mobile Information Systems 2018, 1852861. [Google Scholar] [CrossRef]
18. Xia, Y.; Gong, Y.; Zhang, X.; Bae, H. Y. Location Prediction Based on Variable-order Markov Model and User's Spatio-Temporal Rule. In Proceedings of the IEEE Conference on Information and Communication Technology Convergence, Jeju Island, Korea, Oct. 17-19 2018; pp. 37–40. [Google Scholar]
19. Xu, M.; Han, J. Next Location Recommendation Based on Semantic-Behavior Prediction. In Proceedings of the 5th International Conference on Big Data and Computing, Chengdu, China, May 28-30 2020; pp. 65–73. [Google Scholar]
20. Xu, S.; Cao, J.; Legg, P.; Liu, B.; Li, S. Venue2vec: An Efficient Embedding Model for Fine-Grained User Location Prediction in Geo-Social Networks. IEEE Systems Journal 2019, 14, 1740–1751. [Google Scholar] [CrossRef]
21. Xie, M.; Yin, H.; Wang, H.; Xu, F.; Chen, W.; Wang, S. Learning Graph-based POI Embedding for Location-Based Recommendation. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management, Indianapolis, IN, USA, Oct. 24-28 2016; pp. 15–24. [Google Scholar]
22. Ye, Y.; Zheng, Y.; Chen, Y.; Feng, J.; Xie, X. Mining Individual Life Pattern Based on Location History. In Proceedings of the IEEE Tenth International Conference on Mobile Data Management: Systems, Services, and Middleware, Taipei, Taiwan, May 18-20 2009; pp. 1–10. [Google Scholar]
23. Yao, D.; Zhang, C.; Huang, J.; Bi, J. SERM: A Recurrent Model for Next Location Prediction in Semantic Trajectories. In Proceedings of the ACM on Conference on Information and Knowledge Management, Singapore, Singapore, Nov. 6-10 2017; pp. 2411–2414. [Google Scholar]
24. Ying, J. J. C.; Lee, W. C.; Weng, T. C.; Tseng, V. S. Semantic Trajectory Mining for Location Prediction. In Proceedings of the ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, Chicago, Illinois, USA, Nov. 1-4 2011; pp. 34–43. [Google Scholar]
25. Zhang, X.; Li, B.; Song, C.; Huang, Z.; Li, Y. SASRM: A Semantic and Attention Spatio-Temporal Recurrent Model for Next Location Prediction. In Proceedings of the IEEE International Joint Conference on Neural Networks, Glasgow, United Kingdom, Jul. 19-24 2020; pp. 1–8. [Google Scholar]
26. Zheng, Y.; Xie, X.; Ma, W. Y. GeoLife: A Collaborative Social Networking Service Among User, Location and Trajectory. IEEE Data(base) Engineering Bulletin 2010, 33, 32–39. [Google Scholar]