1. Introduction

2. Related work

3. Method

3.1. Entity extraction

Our entity extraction model adopts the main model structure of RoBERTa-Pointer-Network. First, each character in the input sentence is input into Roberta to obtain the feature vector representation of each character. Then the obtained word vector is input into BiLSTM, and the bidirectional expression of the text sequence containing context information is obtained through the input gate, the forget gate, and the output gate. Finally, the output of the BiLSTM layer is used as the input of the pointer network layer. Each character’s label score in the sequence text is calculated and compared with the standard label, and the optimal label sequence is obtained using the dynamic optimization algorithm. We combine the practice in the field of education to make the following optimization plan.

3.1.1. Incremental Pre-training

Incremental pre-training makes the model more suitable for downstream tasks during the pre-training process by adding task-oriented data. The implementation ideas of incremental pre-training can be roughly divided into data preparation, data processing, model import, training, Etc. In order to improve the performance of the algorithm in downstream tasks, this paper adopts the method of the masked language model. It uses the corpus of robotics to carry out incremental pre-training so that the model can learn the knowledge of the subject domain in advance, which is more suitable for the data of this paper and downstream tasks.

Figure 1. The main architecture of the entity extraction model.

Figure 1. The main architecture of the entity extraction model.

Many experiments have proved that whole word masking is better than random masking because some masking in random masking is predicted based on lexical information, not based on the overall semantics, and we prefer the model to learn semantic information. Therefore, we built a professional vocabulary library in the field of education and used whole word masking for incremental pretraining.

Table 1. An example of the whole word masking.

Table 1. An example of the whole word masking.

Explanation	Sample
sentence	控制机器人末端以一定的夹持力 按既定轨迹移动物体
participle	控制 机器人 末端 以 一定 的 夹持力 按 既定 轨迹 移动 物体
random masking	[Mask]制机器人末端以一定的[Mask]持力 按既定轨迹[Mask]动物体
whole word masking	控制机器人末端以一定的 [Mask] [Mask] [Mask]按既定轨迹移动物体

3.1.2. Focus Loss For education resources

Due to the label extraction process of education resources, some labels are easier to extract, while others are more difficult to extract. For example, in teaching materials, the label of the chapter category is easier to extract, while the labels of sections and subsections are more challenging to identify. According to the difference in recognition difficulty, this paper designs a new type of loss function Focus Loss For education resources(FLFer). It makes the model pay more attention to difficult-to-classify samples during the training process, gives them higher weights, and reduces the impact of easy-to-classify samples on the total loss. The formula is as follows:

alpha = [α1 ,α2 ,α3 ,...,αn], n ∈ labels count

(1)

$$\mathrm{l}\mathrm{o}\mathrm{g}(\mathrm{p}\mathrm{t})=\{\begin{array}{cc}\hat{p}& \mathrm{if}\mathrm{y}=\mathrm{lable},\\ 1-^& \mathrm{otherwise}\end{array}$$

(2)

log(pt) = log(pt) ∗ lapha

(3)

Loss = −(1 − log(pt))γ log(log(pt))

(4)

Among them, alpha is the loss of weight of each label. According to the difficulty of label learning, we set different weights for different labels and calculate the probability corresponding to each weighted label according to the Loss formula. This smoothly adjusts the weight ratio of easy-to-classify samples, balances the weights of difficult-to-classify samples, and reduces the weight of easy-to-classify samples. The model pays more attention to difficult-to-classify samples during training, and the more difficult to learn “specific” labels, the more significant the contribution to the loss value. This loss function can improve the practical effect of sections and subsections. See the ablation experiment section.

3.1.3. Sliding Window

When extracting knowledge points from education resources, there is a problem in that some labels cannot be extracted. Our analysis found that the data cut some entity nodes into two parts, and no data containing complete entities was passed into the model.

Figure 2. Model structure for sliding window.

Figure 2. Model structure for sliding window.

We use the sliding window method in which the window length exceeds the step size, and model training is performed after data processing. The algorithm divides the original data set into a window of 512 characters with a step size of 384 to ensure that all entity characters can be completely present in the model input data. At the same time, it ensures a certain degree of semantic integrity and provides more comprehensive information for the model.

4. Experimental Data

5. Experimental Results Analysis

5.1. Knowledge Point Extraction

The knowledge points in the education resources include the first-level, second-level, and third-level titles of the teaching materials. This article uses the experimental data set and the entity extraction model to extract entities from knowledge points.

The NER ablation experiments are as follows. Based on the original model, we incorporated incremental pre-training in the education field, optimized the loss function, and added a sliding window to make the model more capable of learning the education field’s knowledge characteristics and enhance the model’s generalization ability.

Table 2. Ontology Design of Educational Knowledge Graph.

Table 2. Ontology Design of Educational Knowledge Graph.

Type	Category	Name	Abbreviation	Example
Entity	First-level knowledge point	Chapter	CHP	Chapter Four Sensors
Entity	Second-level knowledge point	Section	SET	4.1Classification of sensors
Entity	Third-level knowledge point	Subsection	SST	4.1.1Built-in sensors
Entity	Finest-grained knowledge point	KeyWord	KEW	Amount of displacement, Linearity
Relation	Hypernym relationship	Contain	COT	<Chapter Four Sensors,4.1Classification of sensors >
Relation	Prerequisite relationship	Preprositon	PPS	<4.1Classification of sensors,5.2Vision system >
Relation	Similar relationship	similarity	SML	<9.1Control technology,9.3 Feedback Control>

Table 3. The details of education resources dataset.

Table 3. The details of education resources dataset.

Label	Train	Val	Test
Chapter	35980	10280	5140
Section	33858	9674	4837
Subsection	22998	6571	3286
Keyword	313969	89541	44770
Prerequisite relation	171256	48930	24465

In order to reflect the advantages of adding modules to the method in this paper, six ablation experiments are set up:

• Ablation focus loss for education resources;
• Ablation incremental pre-training;
• Ablation sliding window;
• Ablation data augmentation;
• Ablation FGM;
• Ablation mixed precision training;

Figure 4. The ablation experiment of KP extraction.

Figure 4. The ablation experiment of KP extraction.

It can be seen that the algorithm model proposed in this paper is more effective than the experimental ablation model on the knowledge point data set, with a maximum increase of 2.7%, 0.9%, and 0.81%, which proves the effectiveness of the algorithm module proposed in this paper. The loss function optimization has achieved good results. The reason for the analysis is that the characteristics of chapters are more pronounced than the other labels, which have general significance and are easy to identify. However, the section and subsection data are long and scattered, and different sections may have similar meanings, which are relatively difficult to identify. Therefore, good results have been achieved using the loss function proposed in this paper.

5.2. Relation Extraction

The goal of subject knowledge map construction is the front and back maps of knowledge points, and the front and back maps of knowledge points can be derived from the skeleton of keywords. Specifically, it is assumed that there are knowledge points KP1 and KP2, the relationship between KP1 and KP2, it can be obtained by incorporating the relationship between keywords.

Table 4. The details of the hyperparameters of the Textbook-Net.

Table 4. The details of the hyperparameters of the Textbook-Net.

Hyperparameters	Textbook-NER	Textbook-KP-RE
Training epoch	5	8
Batch size	32	32
Learning rate	5e-5	5e-5
Loss function	FLFer	Binary Cross Entropy Loss
αchapter	0.8	/
αsection	1.1	/
αsubsection	1.2	/
γ	0.9	/
Max length	512	512
Dropout	0.1	0.15
Mixed Precision	FP16&FP32	FP32

Keywords can represent knowledge points, and several keywords of the subject will appear in the text of specific knowledge points. We serialize the keywords and obtain the keyword graph. The keyword graph ontology structure is keyword A and keyword B and the prerequisite relation, which can be obtained through prior rules. In the catalog of subject textbooks, the order of the knowledge structure is defined, that is, the title order of chapters, sections, and subsections. When keyword A appears in a certain knowledge point, if the keyword appears in the text of a subsequent knowledge point, at the same time, keyword B also exists in the knowledge point, so keyword A is considered the pre-keyword of keyword B.

We use this priority rule to get more than 5,000,000 triple groups of keywords. Then we fuse the keyword graph with the extracted chapters, sections, and subsections and use the related model of knowledge graph embedding to train the embedding of keywords. For example, we used the TransE [2]and GCN [8] models to train the relationship model between keywords and obtained the embedding after integrating the relationship.

We compares the effect experiments of different depth models and text fields as semantic labels. It verifies them on the expert annotation data set in robotics. We use the following knowledge graph embedding model for experimental comparison:

We order the optimal ROBERTa+GCN model as the Textbook-KP-RE model, which is uniformly used for semantic label comparison experiments. We use different corpora as semantic labels of knowledge points for experimental comparison. Among them, the name represents the name text of the knowledge point, the description represents the full description text of the subsection, and the keyword represents the keyword text extracted from the description text.

The results of semantic label experiments show that using the domain-optimized extraction model Textbook-KP-RE combined with knowledge point names and text keywords can achieve the best results, which is higher than other commonly used relationship extraction models.

Table 5. The experimental results (%) of KP relation extraction.

Table 5. The experimental results (%) of KP relation extraction.

T-Encoder	K-Encoder	Precision	Recall	F1 score
BiLSTM	/	84.41	82.14	83.26
BiLSTM	TransE	86.43	85.15	85.78
BERT-base-chinese	/	89.53	88.89	89.2
BERT-base-chinese	TransE	89.23	89.01	89.12
BERT-base-chinese	GCN	91.32	91.1	91.2
ROBERTa-chinese	/	90.2	90.34	90.27
ROBERTa-chinese	TransE	92.58	91.89	92.23
ROBERTa-chinese	GCN	92.68	92.01	92.34

Table 6. The experimental results (%) of semantic label.

Table 6. The experimental results (%) of semantic label.

Semantic Label	Precision	Recall	F1 score
Description	84.17	82.22	83.18
KP-Name&Description	83.51	83.03	83.27
Keyword	89.44	91.19	90.31
KP-Name&Keyword	92.68	91.67	92.34

6. Conclusions

References

1. Meysam Asgari-Chenaghlu,M Reza Feizi-Derakhshi, Leili Farzinvash, MA Balafar, and Cina Motamed, ‘Cwi: a multimodal deep learning approach for named entity recognition from social media using character, word and image features’, Neural Computing and Applications, 1–18, (2022). [CrossRef]
2. MuhaoChen, Yingtao Tian, MohanYang, and Carlo Zaniolo, ‘Multilingual knowledge graph embeddings for cross-lingual knowledge alignment’, arXiv preprint arXiv:1611.03954, (2016). [CrossRef]
3. Hu Y Cheng X, Tang L, ‘Extracting entity and relation of landscape plant’s knowledge based on albert model’, China Journal of Geo- information Science, 23(7), 1208–1220, (7 2021).
4. Tsu-JuiFu, Peng-Hsuan Li, and Wei-Yun Ma, ‘Graphrel: Modeling text as relational graphs for joint entity and relation extraction’,in Proceed- ings of the 57th annual meeting of the association for computational linguistics, pp. 1409–1418, (2019).
5. Sepp Hochreiter and Jürgen Schmidhuber, ‘Long short-term memory’, Neural computation, 9(8), 1735–1780, (1997). [CrossRef]
6. LiJ HuH, ‘Named entity recognition method in educational technology field based on bert’, China Computer Technology and Development, 32(10), 164–168, (10 2022). [CrossRef]
7. Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip, ‘A survey on knowledge graphs: Representation, acquisition, and applications’, IEEE transactions on neural networks and learning systems, 33(2), 494–514, (2021). [CrossRef]
8. Thomas N Kipf and Max Welling, ‘Semi-supervised classification with graph convolutional networks’, arXiv preprint arXiv:1609.02907, (2016). [CrossRef]
9. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner, ‘Gradient-based learning applied to document recognition’, Proceed- ings of the IEEE, 86(11), 2278–2324, (1998). [CrossRef]
10. JDMCKLee and KToutanova, ‘Pre-training of deep bidirectional trans- formers for language understanding’, arXiv preprint arXiv:1810.04805, (2018).
11. Fei Li, Zheng Wang, Siu Cheung Hui, Lejian Liao, Dandan Song, Jing Xu, Guoxiu He, and Meihuizi Jia, ‘Modularized interaction network for named entity recognition’, in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 200–209, (2021).
12. Kaihong Lin, Kehua Miao, Wenxing Hong, and Chaoyi Yuan, ‘Rela- tion extraction based on relation label constraints’, in 2020 IEEE 6th International Conference on Computer and Communications (ICCC), pp. 2166–2170. IEEE, (2020).
13. Pan Liu, Yanming Guo, Fenglei Wang, and GuohuiLi, ‘Chinese named entity recognition: The state of the art’, Neurocomputing, 473, 37–53, (2022). [CrossRef]
14. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoy- anov, ‘Roberta: A robustly optimized bert pretraining approach’, arXiv preprint arXiv:1907.11692, (2019). [CrossRef]
15. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al., ‘Mixed precision training’, arXiv preprint arXiv:1710.03740, (2017). [CrossRef]
16. Takeru Miyato, Andrew M Dai, and Ian Goodfellow, ‘Adversarial train- ing methods for semi-supervised text classification’, arXiv preprint arXiv:1605.07725, (2016). [CrossRef]
17. Marie-Francine Moens, Information extraction: algorithms and prospects in a retrieval context, volume 1, Springer, 2006.
18. Guoshun Nan, Zhijiang Guo, Ivan Sekuli, and Wei Lu, ‘Reasoning with latent structure refinement for document-level relation extraction’, arXiv preprint arXiv:2005.06312, (2020). [CrossRef]
19. S. Ning, F. Teng, and T. R. Li, ‘Multi-channel self-attention mechanism for relation extraction in clinical records’, Chinese Journal of Comput- ers, 43(5), 916–929, (2020).
20. Mark Neumann Peters, Matthew E and Mohit Iyyer, ‘Deep contextual- ized word representations’, in Proceedings of the 2018 Conference of the North American Chapter of the Association for ComputationalLin- guistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2227–2237, (2018).
21. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al., ‘Improving language understanding by generative pre-training’, (2018).
22. Nils Reimers and Iryna Gurevych, ‘Sentence-bert: Sentence embed- dings using siamese bert-networks’, arXiv preprint arXiv:1908.10084, (2019). [CrossRef]
23. FENG Y C REN Y Q, WAN K, ‘The sustainable development of arti- ficial intelligence education——the interpretations and implications of artificial intelligence in educationchallenges and opportunities for sus- tainable development’, Modern Distance Education Research, 31(5), 3–10, (1 2019).
24. Sunil Kumar Sahu, Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou, ‘Inter-sentence relation extraction with document-level graph convolutional neural network’, arXiv preprint arXiv:1906.04684, (2019). [CrossRef]
25. Qingyu Tan, Ruidan He, Lidong Bing, and Hwee Tou Ng, ‘Document-level relation extraction with adaptive focal loss and knowledge distil-lation’, arXiv preprint arXiv:2203.10900, (2022). [CrossRef]
26. Z Yang, Z Dai, Y Yang, J Carbonell, RR Salakhutdinov, and XL-Net Le QV, ‘generalized autoregressive pretraining for language under-standing; 2019′, Preprint at https://arxiv. org/abs/1906.08237 Accessed June, 21, (2021). [CrossRef]
27. Wojciech Zaremba, Ilya Sutskever, and Oriol , ‘Recurrent neural net-work regularization’, arXiv preprint arXiv:1409.2329, (2014). [CrossRef]
28. Ningyu Zhang, Xiang Chen, Xin Xie, Shumin Deng, Chuanqi Tan, Mosha Chen, Fei Huang, Luo Si, and Huajun Chen, ‘Document-level relation extraction as semantic segmentation’, arXiv preprint arXiv:2106.03618, (2021). [CrossRef]
29. SC Zhong, ‘How artificial intelligence supports educational revolution’, China Educational Technology, 3, 17–24, (2020).