3.1. Model Framework

3.2. ASSE

Figure 1. ASSPM. Inputting “东风路文博东路”(Dongfeng Road Wenbo East Road), first, the Address Spatial Semantics Extractor (ASSE) extracts spatial semantic information and generates the Address Element Boundary Matrix (EBM), from which the address elements corresponding to the address text are obtained. Next, using a dictionary (pre-trained word embeddings [23]), all words in the address text are matched to obtain a vocabulary set (including seven elements: “东风”(Dongfeng),“东风路”(Dongfeng Road),“风路”(Feng Road),“路文”(Road Wen),“文博”(Wenbo),“博东”(Bo East) and “东路”(East Road)). The address element set and vocabulary set are then merged, removing words (“路文”(Road Wen)) that do not match in the address element set. Subsequently, two levels of word fusion are performed: one based on EBMSoftLexicon for spatial semantic representation fusion, and the other between spatial semantics and the Transformer layer of the BERT model, thereby constructing the Spatial Semantic Lexicon Enhanced BERT preprocessing model (SSLEBERT). This approach helps integrate and enhance spatial semantic information of address texts. Finally, a Bidirectional Gated Recurrent Unit (BiGRU) is used as the encoding layer to encode the fused spatial semantic representation output by BERT. Simultaneously, a Conditional Random Field (CRF) is employed as the decoding layer to obtain the parsing result of the input address (“东风路文博东路”(Dongfeng Road Wenbo East Road)).

Figure 1. ASSPM. Inputting “东风路文博东路”(Dongfeng Road Wenbo East Road), first, the Address Spatial Semantics Extractor (ASSE) extracts spatial semantic information and generates the Address Element Boundary Matrix (EBM), from which the address elements corresponding to the address text are obtained. Next, using a dictionary (pre-trained word embeddings [23]), all words in the address text are matched to obtain a vocabulary set (including seven elements: “东风”(Dongfeng),“东风路”(Dongfeng Road),“风路”(Feng Road),“路文”(Road Wen),“文博”(Wenbo),“博东”(Bo East) and “东路”(East Road)). The address element set and vocabulary set are then merged, removing words (“路文”(Road Wen)) that do not match in the address element set. Subsequently, two levels of word fusion are performed: one based on EBMSoftLexicon for spatial semantic representation fusion, and the other between spatial semantics and the Transformer layer of the BERT model, thereby constructing the Spatial Semantic Lexicon Enhanced BERT preprocessing model (SSLEBERT). This approach helps integrate and enhance spatial semantic information of address texts. Finally, a Bidirectional Gated Recurrent Unit (BiGRU) is used as the encoding layer to encode the fused spatial semantic representation output by BERT. Simultaneously, a Conditional Random Field (CRF) is employed as the decoding layer to obtain the parsing result of the input address (“东风路文博东路”(Dongfeng Road Wenbo East Road)).

3.3. Spatial Semantic Representation Fusion Module

(1)

Character Representation Layer

For Chinese parsing models based on character-level representations, the input sentence is treated as a sequence of characters, denoted as $s=\{c_1,c_2,\cdots ,c_n\}\in {\mathcal{V}}_c$, where ${\mathcal{V}}_c$ is the character vocabulary [15]. Each character $c_i$ is represented as a vector, as shown in Equation (1):

$${\mathbf{x}}_i^c={\mathbf{e}}^c\left(c_i\right),$$

(1)

where ${\mathbf{e}}^c$ denotes the character embedding lookup table.

Zhang and Yang [14] demonstrated that character bigrams can effectively enhance character representations. Therefore, character bigrams are constructed and concatenated with the vector representation of character $c_i$, as shown in Equation (2):

$${\mathbf{x}}_i^c=\left[{\mathbf{e}}^c\left(c_i\right);{\mathbf{e}}^b\left(c_i,c_{i+1}\right)\right],$$

(2)

where ${\mathbf{e}}^b$ represents the bigram character embedding lookup table. This results in the basic character vector representation. Next, methods to enhance it through the incorporation of dictionary information are discussed.

(2)

Incorporating lexical information

Below we discuss methods to integrate lexical information into ${\mathbf{x}}_i$. To address the limitation of character-based NER models in utilizing lexical information, Ma et al. [15] proposed two approaches: ExSoftword and SoftLexicon. Before delving into these methods, let’s first discuss the representation of words: for any input character sequence $s=\{c_1,c_2,\dots ,c_n\}$, introduce the symbol $w_{i,j}$ to denote the subsequence $\{c_i,c_{i+1},\dots ,c_j\}$. Next, we will discuss three methods for integrating lexical information: ExSoftword, SoftLexicon, and EBMSoftLexicon.

ExSoftword： As an extension of Softword [24,25], ExSoftword not only retains one segmentation result for each character after segmentation but obtains all possible segmentation results from the lexicon, as shown in Equation (3):

$${\mathbf{x}}_j^c\leftarrow \left[{\mathbf{x}}_j^c;{\mathbf{e}}^{seg}\left(segs\left(c_j\right)\right)\right],$$

(3)

where $segs\left(c_j\right)$ represents all classification labels related to $c_j$. For example, for the character “江”(River) in Figure 1, which appears as both a middle and an end character, $segs\left(江\right)=\{M,E\}$. ${\mathbf{e}}^{seg}\left(segs\left(c_j\right)\right)$ is a 5-dimensional multi-hot vector, with each dimension corresponding to {B, M, E, S, O}.

ExSoftword faces two challenges [15]: 1) It cannot utilize pre-trained word models. 2) It suffers from missing matching information, leading to multiple matching results.

SoftLexicon：To address the aforementioned issues, SoftLexicon [15] offers a solution that involves three primary steps:

Firstly, in the initial step, the matched vocabulary is categorized. To retain segmentation information, each character matched to a word is classified into {B, M, E, S}. For each character $c_i$, the four sets are defined as follows in Equation (4):

$$\begin{array}{cc}\hfill & B\left(c_i\right)=\left\{w_{i,k},\forall w_{i,k}\in L,i<k\le n\right\},\hfill \\ \hfill & M\left(c_i\right)=\left\{w_{j,k},\forall w_{j,k}\in L,1\le j<i<k\le n\right\},\hfill \\ \hfill & E\left(c_i\right)=\left\{w_{j,i},\forall w_{j,i}\in L,1\le j<i\right\},\hfill \\ \hfill & S\left(c_i\right)=\left\{c_i,\exists c_i\in L\right\},\hfill \end{array}$$

(4)

where L denotes the dictionary used in this study. If no vocabulary information matches, it is replaced with "None". This approach embeds words while mitigating information loss.

Secondly, the sets {B, M, E, S} of vocabulary vectors are compressed into fixed-dimensional vectors. After obtaining the sets B, M, E, S for each character, they are compressed into fixed-dimensional vectors in two ways. The first approach is to directly average them, as shown in Equation (5):

$$v^u\left(U\right)=\frac{1}{\left|U\right|}\sum _{w\in U}{e}^w\left(w\right),$$

(5)

where U represents the set of vocabulary, i.e., the set of vocabulary, and $e^w\left(w\right)$ represents the word embedding.

At the same time, in order to maintain computational efficiency, dynamic weighted algorithms such as attention mechanisms have not been adopted. Ma et al. [15] suggested using frequency to represent the weight of each word. Since the frequency of a word can be obtained offline as a static value, this significantly improves the efficiency of calculating the weight of each word. The literature confirmed that the compression method implemented by Equation (5) is not ideal.

Next, the second approach is discussed. Specifically, let $z\left(w\right)$ represent the frequency at which the word w appears in the dictionary. Thus, the weighted representation of the set U can be obtained, as shown in Equation (6):

$$v^u\left(U\right)=\frac{4}{Z}z\left(w\right)e^w\left(w\right),$$

(6)

where $Z={\sum }_{w\in B\cup M\cup E\cup S}z\left(w\right)$. If w is covered by another subsequence matching the dictionary, the frequency of w will not increase. This prevents the problem where the frequency of shorter words is always smaller than that of longer words covering them.

Finally, the third step involves merging character representations. Once the fixed-dimensional representations of the four vocabulary sets are obtained, they can be combined into a fixed-dimensional feature, as shown in Equation (7):

$$e^u(B,M,E,S)=[v^u\left(B\right);v^u\left(M\right);v^u\left(E\right);v^u\left(S\right)],$$

(7)

where $v^u$ is the weighted function provided by Equation (6). Thus, Equation (7) yields the SoftLexicon character representation, as shown in Equation (8):

$$x^c\leftarrow [x^c;e^u(B,M,E,S)],$$

(8)

EBMSoftLexicon：While SoftLexicon has demonstrated significant performance, particularly showcasing its potential in low-resource scenarios, for Chinese address parsing tasks, this approach introduces some unnecessary lexical information. For instance, in the context of “东风路”(Dongfeng Road) depicted in Figure 1, the lexical information for “东风”(Dongfeng) could be filtered out entirely due to the prominent tail feature of the character “路”(road). Similarly, “文博”(Wenbo) and “博东”(Bo East) serve as examples of irrelevant “interference” information that does not need consideration. To address the challenges identified, an enhanced method called EBMSoftLexicon is proposed to integrate lexical information more effectively, particularly for Chinese address parsing tasks. This method combines the SoftLexicon approach with Address Element Boundary Matrix (EBM) information derived from ASSE. This integration aims to refine or enhance SoftLexicon by incorporating boundary information, confidence scores, and evaluation metrics from EBM.

Specifically, the approach involves calculating a weighted vector representation $E\left(w\right)$ for each vocabulary word w, as defined in Equation (9):

$$E\left(w\right)=\sum _{w\in \mathbf{ADD}}scor{e}^w·C^w,$$

(9)

where $scor{e}^w$ and $C^w$ are determined by Equations (10) and (11), respectively.

$$score=\frac{log\left(score\right)}{log\left(MA{X}_{score}\right)},$$

(10)

$$C=XGBoostCC(\overrightarrow{x};P)=xgb.predict(D\left(\overrightarrow{x}\right),xgb\left(P\right)),$$

(11)

In equation (11), $\overrightarrow{x}$ represents an 8-dimensional input vector, specifically the address state, composed of address element type, transition probability, length information, matching start and end positions, regular expression score, matching length, and a numeric indicator (1 for Chinese and Arabic numerals, 0 otherwise). P denotes the parameters of the XGBoost, with the experimental section detailing the configuration of these parameters to obtain P. $D\left(\overrightarrow{x}\right)$ denotes the function that converts the input vector $\overrightarrow{x}$ into a DMatrix data structure. $xgb\left(P\right)$ refers to the XGBoost trained using the parameters P, and $xgb.predict$ is the function used to make predictions with the trained model.

$\mathbf{ADD}$ denotes the annotated dataset. Correspondingly, let $E={\sum }_{w\in B\cup M\cup E\cup S}E\left(w\right)$, which allows us to obtain the weighted representation of the word set U in EBMSoftLexicon, as shown in equation (12):

$$V^u\left(U\right)=\left\{\begin{array}{cc}\frac{4}{E}E\left(w\right)e^w\left(w\right),\hfill & \mathrm{if}\exists \left(EB{M}^w\right),\hfill \\ v^u\left(U\right),\hfill & \mathrm{else}.\hfill \end{array}\right.$$

(12)

Here, $\exists \left(EB{M}^w\right)$ indicates that the EBM score surpasses the threshold, allowing for enhanced embedding using $E\left(w\right)$ and $e^w\left(w\right)$. If the condition is not met (else case), or for non-Chinese address data where EBM information isn’t applicable, a fallback to SoftLexicon embedding $v^u\left(U\right)$ occurs.

EBMSoftLexicon is obtained offline through ASSE. This method first uses ASSE to extract all Chinese addresses’ EBM from the annotated dataset. This extraction differs slightly from the original data extraction because basic information can be obtained directly from the annotation information, such as the matching targets and address element boundary types in equation (13). This significantly reduces the complexity of solving and allows evaluation information to be obtained through the evaluation mechanism in the extractor. Then, equation (9) is used to obtain the weights of all vocabulary in the current dataset. It is important to note that vocabulary exists in two forms: complete address elements represented by EBM column vectors, and boundary information contained in EBM. In cases where both forms exist, preference is given to the former due to its inclusion of the latter.

$$E_i=P\left(\Psi (add,T_j,AEBRL)\right),$$

(13)

Here, $add$ represents the address text, $T_j$ denotes a specific address element type, and $\Psi$ extracts boundary information using $T_j$ and $AEBRL$. The function P combines the results of $\Psi$ into $E_i$, which populates the first seven dimensions of $E_{i,j}^k={\left(i,j,k,\left[k\right],W_k,a,b,C\right)}^T$. The $\Psi$ function is detailed in Algorithm 1:

Finally, substituting equation (12) into equations (7) and (8), the EBMSoftLexicon character representation is obtained as shown in equation (14):

$$x^c\leftarrow [x^c;[V^u\left(B\right);V^u\left(M\right);V^u\left(E\right);V^u\left(S\right)]],$$

(14)

In summary, EBMSoftLexicon not only avoids introducing unnecessary or "noise" information directly from dictionary lookups, but also leverages boundary and spatial distribution information effectively. By obtaining vocabulary information from ASSE, which includes evaluation scores and confidence levels, when EBM is not obtained or the obtained EBM has insufficient scores (corresponding to the $else$ case in equation (12)), the solution follows equation (6). Therefore, the EBMSoftLexicon represented by equation (14) is compatible with SoftLexicon, contributing to enhanced model stability. This approach achieves a blended representation of spatial semantics and deep parsing models through enhanced embedding layers.

3.4. Spatial Semantic Lexicon Enhanced BERT Module

(1)

Enhanced Spatial Semantics of Character-Word Pairs

A Chinese sentence is typically represented as a sequence of characters, which only captures character-level features. To incorporate lexical information, the character sequence is expanded into a sequence of character-word pairs. Given a Chinese dictionary $\mathbf{D}$ and a sentence ${\mathbf{s}}_{\mathbf{c}}=\{c_1,c_2,...,c_n\}$ consisting of n characters, potential words in the sentence, denoted as $WS$, are identified by matching the character sequence with $\mathbf{D}$. Specifically, a Trie tree is constructed based on $\mathbf{D}$, and all possible character subsequences in the sentence are matched against this Trie tree to obtain all potential words, denoted as $D_s$. To enhance spatial semantics representation, lexical elements from the dictionary and address features extracted by the ASSE are utilized. ${\mathbf{SS}}_{\mathbf{s}}$ denotes the vocabulary sequence obtained by ASSE, referred to as spatial vocabulary sequence. For each address, each corresponding $WS$ is sequentially searched within ${\mathbf{SS}}_{\mathbf{s}}$ for substring matches. Words in $WS$ that cannot be found in ${\mathbf{SS}}_{\mathbf{s}}$ are removed, and finally, the union operation is applied between $WS$ and ${\mathbf{SS}}_{\mathbf{s}}$. The spatial semantics-enhanced character-word pairs corresponding to the addresses in Figure 1 are illustrated in Figure 3.

(2)

Lexicon Adapter

Each position in every sentence contains two distinct types of information: character-level and word-level features. Consistent with existing hybrid model approaches, this study aims to effectively integrate lexicon features with BERT. The Lexicon Adapter (LA) [20], depicted in Figure 4, illustrates this integration process.

The Lexicon Adapter accepts two inputs: character vectors and their corresponding word embeddings. At the i-th position in the character-word pair sequence, inputs are represented as $(h_i^c,x_i^{ws})$, where $h_i^c$ is the character vector obtained from a Transformer layer of BERT, and $x_i^{ws}=\{x_{i1}^w,x_{i2}^w,...,x_{im}^w\}$ is a set of word embeddings. The j-th word in $x_i^{ws}$ is represented as shown in Equation (15):

$$x_{ij}^w={\mathbf{e}}^w\left(w_{ij}\right),$$

(15)

where ${\mathbf{e}}^w$ is a pre-trained word embedding lookup table and $w_{ij}$ is the j-th word in $w{s}_i$. It’s important to note that the Address Semantic Extractor (ASSE) may yield words in ${\mathbf{SS}}_{\mathbf{s}}$ that are not in the vocabulary. In such cases, these words are split into multiple components and their corresponding word vectors are looked up individually using Equation (12).

To ensure compatibility between these two representations, a nonlinear transformation is applied to the word vectors, as shown in Equation (16):

$$v_{ij}^w={\mathbf{W}}_2\left(\tanh ({\mathbf{W}}_1{x}_{ij}^w+{\mathbf{b}}_1)\right)+{\mathbf{b}}_2,$$

(16)

where ${\mathbf{W}}_1$ is a $d_c\times d_w$ matrix, ${\mathbf{W}}_2$ is a $d_c\times d_c$ matrix, and ${\mathbf{b}}_1$ and ${\mathbf{b}}_2$ are scalar biases. $d_w$ and $d_c$ represent the dimensions of word embeddings and BERT’s hidden layers, respectively.

As depicted in Figure 3, each Chinese character corresponds to multiple words. However, each word contributes differently to each task. To select the most relevant words from all matched words, a character-to-word attention mechanism is introduced. All $v_{ij}^w$ assigned to the i-th character are represented as $V_i=(v_{i1}^w,...,v_{im}^w)$, where m is the total number of assigned words. The relevance of each word can be calculated using Equation (17):

$${\mathbf{a}}_i=\mathrm{softmax}\left(h_i^c{\mathbf{W}}_{attn}{V_i}^T\right),$$

(17)

where ${\mathbf{W}}_{attn}$ is the weight matrix for bilinear attention. Thus, the weighted sum of all words is obtained using Equation (18):

$$z_i^w=\sum _{j=1}^m{a}_{ij}{v}_{ij}^w,$$

(18)

Finally, the weighted lexicon information is injected into the character vector using Equation (19):

$${\tilde{h}}_i=h_i^c+z_i^w,$$

(19)

(3)

Lexicon-Enhanced BERT

Lexicon-Enhanced BERT (LEBERT) combines the Lexicon Adapter (LA) with BERT, where LA is applied to a specific layer of BERT to incorporate external lexicon knowledge into the model. Given a Chinese sentence ${\mathbf{s}}_c=\{c_1,c_2,...,c_n\}$, we construct the corresponding character-word pair sequence ${\mathbf{s}}_{cw}=\{(c_1,w{s}_1),(c_2,w{s}_2),...,(c_n,w{s}_n)\}$.

Firstly, the characters $\{c_1,c_2,...,c_n\}$ are input into the input embeddings, which include token, segment, and positional embeddings, resulting in $E=\{e_1,e_2,...,e_n\}$. These embeddings E are then passed through the transformer encoder, where each transformer layer operation is defined as in Equation (20):

$$\begin{array}{cc}\hfill G& =\mathrm{LN}(H^{l-1}+\mathrm{MultiHeadAttention}\left(H^{l-1}\right))\hfill \\ \hfill H^l& =\mathrm{LN}(G+\mathrm{FFN}(G\left)\right)\hfill \end{array},$$

(20)

where $H^l=\{h_1^l,h_2^l,...,h_n^l\}$ represents the output of the l-th layer, $H^0=E$; LN denotes layer normalization; MultiHeadAttention refers to the multi-head attention mechanism; FFN denotes a two-layer feed-forward network with ReLU as the hidden activation function.

To inject lexicon information between the k-th and $(k+1)$-th Transformer layers, we first obtain the output $H^k=\{h_1^k,h_2^k,...,h_n^k\}$ after passing through k consecutive Transformer layers. Then, each pair $(h_i^k,x_i^{ws})$ is processed through the Lexicon Adapter, transforming the i-th pair into ${\tilde{h}}_i^k$, as shown in Equation (21):

$${\tilde{h}}_i^k=\mathrm{LA}(h_i^k,x_i^{ws}),$$

(21)

Given that BERT consists of $L=12$ Transformer layers, ${\tilde{H}}^k=\{{\tilde{h}}_1^k,{\tilde{h}}_2^k,...,{\tilde{h}}_n^k\}$ is then input into the remaining $(L-k)$ Transformers, resulting in the output $H^L$ of the L-th Transformer for sequence labeling tasks.

3.5. Encoding-Decoding and Loss Function

4.1. Dataset Processing

4.2. Experimental Setup and Evaluation Metrics

4.3. Comparison with Baseline Methods

4.4. Ablation Experiments

4.5. Results Discussion