1. Introduction

2. Related Work

3. The SFRARE Model

4. Experiments

4.1. Tibet Railway Traffic text Dataset

Due to the unavailability of a publicly accessible text dataset specifically focused on railway traffic in Tibet, we borrow ideas from Zhang and Rong’s works. Specifically, we utilized the history of railway construction in Tibet as an index to gather relevant textual data from the internet. For instance, we extracted texts containing knowledge related to railway traffic from the introduction of Qinghai-Tibet Railway on Baidu Baike (a Chinese online encyclopedia). Subsequently, we manually annotated the relational triplets present in these texts. These annotated textual data were then utilized to train our model, as illustrated in Figure 3.

In addition, we conducted searches using each railway station in Tibet as the departure and arrival stations to find train numbers. We then utilized the relevant information associated with these train numbers to create textual data. This portion of the text was also manually annotated with relational triplets for training the model in relation extraction tasks. We collected a total of 2056 textual data samples related to railway traffic in Tibet. These samples were used to construct a small Chinese text dataset specifically for the task of relation extraction in Tibet’s railway traffic domain. The constructed dataset was further divided into train set, validation set, and test set, with a ratio of 8:1:1 respectively. Details of the dataset division are presented in Table 1.

Table 1. Data Statistics for Tibet Railway Traffic text Dataset.

Table 1. Data Statistics for Tibet Railway Traffic text Dataset.

Dataset	Train	Valid	Test	ALL
Tibet Railway Traffic text Dataset	1645	205	206	2056

Through the exploration of railway traffic knowledge in Tibet, we have identified a total of 16 types of relations. These relation types were defined in Chinese originally, and the translation in English is shown in Table 2.

Table 2. Relation types of Tibet Railway Traffic text Dataset.

Table 2. Relation types of Tibet Railway Traffic text Dataset.

Relation Type	The number of triplets	Relation Type	The number of triplets
railway station-be located in-location	205	railway station-cover an area of-area	163
railway-alternative name-name	23	railway-construction date-date	35
railway-finish date-date	63	railway-opening date-date	32
railway-length of railway-length	22	railway-width of railway-width	17
railway-height of railway-elevation	26	railway-belong to-railway	46
train-departure station-railway station	54	train-terminal station-railway station	54
train-departure time-time	54	train-journey time-time	54
constructors-construct-railway	468	passenger-take the train-train	762
The number of all triplets		2078

As depicted in Table 2, we have defined 16 relation types for the dataset that we constructed. The table displays the format of the relational triplets formed by each relation type, along with their corresponding subject and object. For instance, the relation type ’railway-alternative name-name’ signifies that when the subject is a railway and the object is a name, there might exist an ’alternative name’ relation between them. Subsequently, we conducted a count of the number of relational triplets for each relation type and determined a total of 2078 relational triplets within the dataset.

4.2. Public Datasets

In this paper, English public datasets NYT[19] and WebNLG are used to verify the generalization ability of the proposed model. The NYT dataset is a publicly available English dataset constructed using a remote supervision method. It consists of 24 relations for relation extraction tasks. On the other hand, the WebNLG dataset was initially created for the natural language generation task but has been adapted for the relation extraction task. It contains a broader range of relation types, encompassing a total of 246 different relations. The relation types present in the two datasets are detailed in Table 3.

Table 3. Data Statistics for NYT Dataset and WebNLG Dataset

Table 3. Data Statistics for NYT Dataset and WebNLG Dataset

Category	NYT			WebNLG
	Train	Valid	Test	Train	Valid	Test
All	56195	4999	5000	5019	500	703

4.3. Experimental setup

The specific experimental settings of this paper are shown in Table 4.

Table 4. Experimental details

Table 4. Experimental details

Items	Specific setting
Word embedding layer	bert_base&bert_chinese
Recognition threshold	0.5
Hidden layer dimension of LSTM	768
Number of heads of attention mechanism	num-rels
Learning rate	1e-5
Batch-size	16
Dropout	0.2
Experimental device	NVIDIA GeForce RTX 3090(24G)

We employe the BERT model developed by Google as the word embedding layer. Specifically, for the experiment conducted on the Tibet Railway Traffic text Dataset, we utilize the bert_chinese version of BERT. For the experiments conducted on the NYT and WebNLG datasets, we use the bert_base version of BERT. The recognition thresholds for both the subject recognizer and the relation-object recognizer are set to 0.5. The hidden layer dimension of the LSTM is set to 768. Additionally, the number of heads in the two-layer multi-head attention mechanism is set to the number of relations defined by each respective dataset. During the training process, we employed the Adam optimizer with a learning rate of 1e-5 and a batch size of 16. A dropout rate of 0.5 was applied. The experimental device used was an NVIDIA GeForce RTX 3090 graphics card with 24GB of video memory. To evaluate the extraction results, we calculated the accuracy rate, recall rate, and F1-score, as shown in Equations (15) to (17).

$$Pre=\frac{TT}{PT}$$

(15)

$$Rec=\frac{TT}{GT}$$

(16)

$$F1=\frac{2\ast (Pre\ast Rec)}{Pre+Rec}$$

(17)

where $TT$ is the number of correct triplets predicted, $PT$ is the number of all predicted triplets, and $GT$ is the number of actually correct triplets.

4.4. Experimental result

4.4.1. Experimental results on Tibet Railway Traffic text Dataset

Due to the limitation of experimental equipment, we could not solve the problem that some models using bert_chinese as the word embedding layer exceeded the video memory during training. In this experimental part, we only chose the baseline model CasRel as the comparison model.

• CasRel : A novel cascade binary tagging framework for relational triplet extraction proposed by Wei et al.

The experimental results on Tibet Railway Traffic text Dataset are shown in Table 5.

Table 5. Experimental results on Xizang Railway Traffic Dataset.

Table 5. Experimental results on Xizang Railway Traffic Dataset.

Model	Tibet Railway Traffic text Dataset
	Pre	Rec	F1
CasRel	93.1%	91.9%	92.5%
SFRARE	93.7%	93.0%	93.3%

According to the experimental results presented in Table 5, the SFRARE model achieves a higher accuracy rate compared to CasRel, with an improvement of 0.6%. This suggests that the SFRARE model mitigates the impact of error propagation to some extent in comparison to CasRel. The recall rate of SFRARE is also 1.1% higher than CasRel, indicating that SFRARE better captures more relational triples in the Tibet Railway Traffic text Dataset. This implies that SFRARE performs better on this small dataset. Lastly, the F1-score achieved by SFRARE is 0.8% higher than CasRel, further supporting the conclusion that SFRARE outperforms CasRel on the Tibet Railway Traffic text Dataset.

4.4.2. Experimental results on public datasets

To assess the generalization capability of SFRARE, this paper includes several state-of-the-art relation extraction models that have previously published experimental results on two public datasets: NYT and WebNLG. In the experimental section, CasRel is selected as the baseline model for comparison against these representative models.

• CopyR${}_{OneDecoder}$[20] : An end-to-end relation extraction model with replication mechanism proposed by Zeng et al., a single decoder is used at the decoding layer.
• CopyR${}_{MultiDecoder}$ : An end-to-end relational extraction model with replication mechanism proposed by Zeng et al., multiple decoders are used at the decoding layer.
• GraphRel${}_{1p}$[21] : The first phase of relational extraction model based on relational graph structure proposed by Tsu-Jui Fu et al.
• GraphRel${}_{2p}$ : The second phase of relational extraction model based on relational graph structure proposed by Tsu-Jui Fu et al.
• CopyR${}_{RL}$[22] : Zeng et al. proposed an end-to-end relation extraction model based on CopyR that applies reinforcement learning to the generation of relational triplets.
• CasRel : A novel cascade binary tagging framework for relational triplet extraction proposed by Wei et al.

The experimental results on the public dataset are shown in Table 6.

Table 6. Experimental results on NYT and WebNLG

Table 6. Experimental results on NYT and WebNLG

Model	NYT			WebNLG
	Pre	Rec	F1	Pre	Rec	F1
CopyR${}_{{}_{OneDecoder}}$	59.4%	53.1%	56.0%	32.2%	28.9%	30.5%
CopyR${}_{{}_{MultiDecoder}}$	61.0%	56.6%	58.7%	37.7%	36.4%	37.1%
GraphRel${}_{1p}$	62.9%	57.3%	60.0%	42.3%	39.2%	40.7%
GraphRel${}_{2p}$	63.9%	60.0%	61.9%	44.7%	41.1%	42.9%
CopyR${}_{RL}$	77.9%	67.2%	72.1%	63.3%	59.9%	61.6%
CasRel	89.7%	89.5%	89.6%	93.4%	90.1%	91.8%
SFRARE	90.5%	91.7%	91.1%	92.6%	92.7%	92.6%

Table 6 shows that SFRARE outperforms the baseline model on both the NYT and WebNLG datasets. On the NYT dataset, SFRARE achieves a precision rate of 90.5%, a recall rate of 91.7%, and an F1-score of 91.1%. These scores are 0.8%, 2.2%, and 1.5% higher than those of the baseline model CasRel. Similarly, on the WebNLG dataset, SFRARE achieves a precision rate of 92.6%, a recall rate of 92.7%, and an F1-score of 92.6%. Even the precision rate of SFRARE is slightly lower than that of CasRel, it improves the recall and F1-score by 2.6% and 0.8%, respectively. The experimental results demonstrate that the SFRARE model exhibits favorable generalization ability and surpasses mainstream models in terms of extraction performance.

4.4.3. Ablation experiment

To examine the influence of subject feature enhancement and relational attention enhancement on the extraction performance of the SFRARE model, we conducted ablation experiments on the NYT and WebNLG datasets. The results of these experiments are presented in Table 4. In the table, SFRARE${}_{{}_{SubF}}$ represents the variant of the SFRARE model that only incorporates enhanced subject features, while SFRARE${}_{{}_{RelA}}$ denotes the variant that solely incorporates enhanced relational attention. On the other hand, SFRARE refers to the complete model that combines both subject feature enhancement and relational attention enhancement.

Table 7. The results of the ablation experiments on NYT and WebNLG.

Table 7. The results of the ablation experiments on NYT and WebNLG.

Methods	NYT			WebNLG
	Pre	Rec	F1	Pre	Rec	F1
CasRel	89.7%	89.5%	89.6%	93.4%	90.1%	91.8%
SFRARE${}_{{}_{SubF}}$	90.1%	89.8%	89.9%	92.2%	91.8%	92.0%
SFRARE${}_{{}_{RelA}}$	89.9%	91.0%	90.5%	93.7%	90.7%	92.2%
SFRARE	90.5%	91.7%	91.1%	92.6%	92.7%	92.6%

The experimental results indicate that enhancing subject features leads to an increase in the F1-scores achieved by SFRARE on both datasets. Specifically, compared to the baseline model, SFRARE achieves a 0.3% and 0.2% improvement in F1-score on the two datasets, respectively. Similarly, enhancing relational attention also contributes to improved performance. In comparison to the baseline model, SFRARE demonstrates an F1-score increase of 0.9% and 0.4% on the two datasets, respectively. When both subject features and relational attention are enhanced, the overall performance improvement is even more significant. SFRARE achieves an increase of 1.5% and 0.8% in F1-score on the two datasets, respectively. The results from the ablation experiments confirm that enhancing subject features and relational attention has a positive impact on the SFRARE model.

Based on the experimental results, it can be concluded that the SFRARE model surpasses the baseline CasRel and mainstream extraction models, on both the Chinese dataset (the Tibet Railway Traffic text Dataset) and two public English datasets (NYT and WebNLG). These findings demonstrate the effectiveness and generalization capability of the SFRARE model for knowledge graph construction in the field of railway traffic in Tibet. The improved performance of the SFRARE can be attributed to several factors. Firstly, the subject features enhancement module enhances the semantic features of the subject and the input text through LSTM. This provides more accurate information to the relation-object recognizer, thereby mitigating the effects of error propagation caused by inaccurate subject feature recognition. Furthermore, the relational attention enhancement module calculates the weight of each defined relation using two multi-attention mechanisms. This approach allows the model to focus more attention on relations related to the subject, thereby improving the performance of the model in extracting relation triplets. Moreover, this attention mechanism enables the model to achieve good performance even with small training datasets. Overall, the enhancements made in the SFRARE model contribute to its superior performance in relation extraction tasks, making it a suitable choice for knowledge graph construction in the field of railway traffic in Tibet.

5. Conclusions

References

1. Miller, S.; Fox, H.; Ramshaw, L. ; Weischedel, R A Novel Use of Statistical Parsing to Extract Information from Text. In Proceedings of the 6th Applied Natural Language Processing Conference, Seattle, Washington, USA, 29 April–4 May 2000; pp. 226–233. [Google Scholar]
2. Chen, Y.; Zheng, D.; Zhao, T. Chinese relation extraction based on Deep Belief Nets. Journal of Software 2012, 23, 2572–2585. [Google Scholar] [CrossRef]
3. E, H.; Zhang, W.; Xiao, S.; Cheng, R.; Hu, Y.; Zhou, X.; Niu, P. Joint entity relationship extraction based on deep learning. Journal of Software 2019, 30, 1793–1818. [Google Scholar]
4. Zhang, Y. A historical survey of the construction of Xizang Railway. China’s Borderland History and Geography Studies 2015, 3, 32–43. [Google Scholar]
5. Rong, Z. The role of Qinghai-Xizang Railway in the evolution of temporal and spatial structure of traffic in Xizang. China Tibetology 2016, 2, 62–71. [Google Scholar]
6. Zhang, S.; Wang, X.; Chen, Z.; Wang, L.; Xu, D.; Jia, Y. Survey of Supervised Joint Entity Relation Extraction Methods. Journal of Frontiers of Computer Science and Technology 2022, 16, 713–733. [Google Scholar]
7. Miwa, M.; Bansal, M. End-to-end relation extraction using LSTMs on sequences and tree structures. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany, 7–12 August 2016; pp. 1105–1116. [Google Scholar]
8. Tan, Z.; Zhao, X.; Wang, W.; Xiao, W. Jointly extracting multiple triplets with multilayer translation constraints. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, Honolulu, Hawaii, USA, 27 January–1 February 2019; pp. 7080–7087. [Google Scholar]
9. Liu, J.; Chen, S.; Wang, B.; Zhang, J.; Xu, T. Attention as relation:learning supervised multi-head self-attention for relation extraction. In Proceedings of the 29th International Joint Conference on Artificial Intelligence, Yokohama, Japan, 7–15 January 2021; pp. 3787–3793. [Google Scholar]
10. Sun, K.; Zhang, R.; Mensah, S.; Mao, Y.; Liu, X. Recurrent interaction network for jointly extracting entities and classifying relations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online, 16–20 November 2020; pp. 3722–3732. [Google Scholar]
11. Wei, Z.; Su, j.; Wang, Y.; Tian, Y.; Chang, Y. A Novel Cascade Binary Tagging Framework for Relational Triple Extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Seattle, Washington, USA, 5–10 July 2020; pp. 1476–1488. [Google Scholar]
12. Wang, Y.; Yu, B.; Zhang, Y.; Liu, T.; Zhu, H.; Sun, L. TPLinker: Single-stage Joint Extraction of Entities and Relations Through Token Pair Linking. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online), 8–13 December 2020; pp. 1572–1582. [Google Scholar]
13. Wang, J.; Lu, W. Two are better than one: joint entity and relation extraction with table-sequence encoders. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, Online, 16–20 November 2020; pp. 1706–1721. [Google Scholar]
14. Zheng, S.; Wang, F.; Bao, H.; Hao, Y.; Zhou, P.; Xu, B. Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, 30 July-4 August 2017; pp. 1227–1236. [Google Scholar]
15. Dai, D.; Xiao, X.; Lyu, Y.; Dou, S.; Wang, H. Joint extraction of entities and overlapping relations using position-attentive sequence labeling. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, Honolulu, Hawaii, USA, 27 January-1 February 2019; pp. 6300–6308. [Google Scholar]
16. Sui, D.; Zeng, X.; Chen, Y.; Liu, K.; Zhao, J. Joint Entity and Relation Extraction With Set Prediction Networks. IEEE Transactions on Neural Networks and Learning Systems 2020, PP. [Google Scholar] [CrossRef] [PubMed]
17. Vaswani, A.; Shazeer, N.; Parmer, N.; Uszkoreit, J.; Jones, L.; Gomez, N.A.; Kaiser, L.; Polosukhin, l. Attention Is All You Need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, California, USA, 4-9 December 2017; pp. 5998–6008. [Google Scholar]
18. Jacob, D.; Chang, M.W.; Kenton, L.; Kristina, T. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Volume 1 (Long and Short Papers), Minneapolis, USA, 2-7 June 2019; pp. 4171–4186. [Google Scholar]
19. Ren, X.; Wu, Z.; He, W.; Meng, Q.; Voss, R.C.; Ji, H.; Abdelzaher, F.T.; Han, J. CoType: joint extraction of typed entities and relations with knowledge bases. In Proceedings of the 26th International Conference on World Wide Web, Perth, Australia, 3-7 April 2017; pp. 1015–1024. [Google Scholar]
20. Zeng, X.; Zeng, D.; He, S.; Liu, K.; Zhao, J. Extracting relational facts by an end-to-end neural model with copy mechanism. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume1: Long Papers), Melbourne, Australia, 15-20 July 2018; pp. 506–514. [Google Scholar]
21. Fu, T.J.; Li, P.H.; Ma, W.Y. Modeling text as relational graphs for joint entity and relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 30 July-4 August; pp. 1409–1418.
22. Zeng, X.; He, S.; Zeng, D.; Liu, K.; Liu, S.; Zhao, J. Learning the extraction order of multiple relational facts in a sentence with reinforcement learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 3-7 November 2019; pp. 367–377. [Google Scholar]