1. Introduction

2. State-of-the-Art

3. Datasets and Preprocessing Methodology

3.1. Bibliographic Data

The Giant corpus [18] was selected as the training dataset for its breadth and diversity in bibliographic references. This comprehensive dataset encompasses a vast array of citation styles and document types, presenting a rich tapestry of bibliographic data that spans across numerous academic disciplines and publication formats. By training models on such a heterogeneous dataset, we aim to cultivate an architecture that learns the intricate patterns and variations inherent in bibliographic references and possesses the versatility to adapt to the myriad ways academic information can be structured. As for the evaluation of the model’s efficiency on an independent dataset, the CORA corpus [19] was used, distinguished by its detailed structure and frequent use as a benchmark in reference segmentation studies [13,14,20].

3.1.1. Giant Corpus

The Giant corpus contains 991,411,110 records, divided into 1568 different citation styles and encompasses 24 types of documents1 [18]. Each record has the following structure:

• {
•   "doi": "10.2307/2177340",
•   "articleType": 3,
•   "citationStyle": 0,
•   "citationStringAnnotated": "<author><family>Ritchie</family>,
•   <given>E.</given> and <family>Powell</family>, <given>Elmer Ellsworth
•   </given></author>(<issued><year>1907</year></issued>) <title>Spinoza
•   and Religion.</title> <container-title>The Philosophical Review
•   </container-title>, <volume>16</volume>(<issue>3</issue>), p.
•   <page>339</page>. [online] Available from: <URL>http://dx.doi.
•   org/10.2307/2177340</URL>"
• }

Each field contains the following information:

• doi: Digital Object Identifier (DOI) is an unique identifier of the document it represents2.
• articleType: The identifier representing the type of document (thesis, article, book, etc.).
• citationStyle: The identifier representing the citation style (APA, Harvard, IEEE, etc).
• citationStringAnnotated: The annotated reference string.

The reference string comes annotated with the following eXtensible Markup Language (XML) structure:

The "citationStringAnnotated" in the Giant corpus provides an annotated reference string for each record, using XML-like tags to mark different bibliographic elements such as authors, titles, and publication details. This structured annotation facilitates the precise extraction of bibliographic information, crucial for training models to accurately segment and understand the various components of academic references.

3.1.2. CORA Corpus

It is a human-annotated corpus of 1877 bibliographic reference strings with a variety of formats and styles, including magazine preprints, conference papers, and technical reports3 [19].

3.2. Preprocessing

The necessity to reprocess the data stemmed from a strategic decision aimed at reducing the variability and computational complexity inherent in the original dataset. Given the vast diversity and multitude of citation styles and document types in the Giant corpus, the initial data presented a significant challenge in terms of model training and evaluation. The variety in formatting and structuring of bibliographic references, while valuable for understanding real-world application scenarios, introduced a level of complexity that could potentially hinder the model’s ability to learn consistent patterns and generalize across unseen data.

By simplifying the dataset and streamlining the annotation structure, the goal was to distill the essential bibliographic components that are most relevant for the task of reference segmentation. This preprocessing step was designed to minimize extraneous variability that does not contribute to the core objective of the study, thereby allowing the models to focus on learning the fundamental syntactic and structural features of bibliographic references. In doing so, the computational demands on the models were significantly reduced, enhancing their efficiency and effectiveness in segmenting references. Preprocessing not only facilitates a more focused and efficient training process but also aims to improve the models’ generalizability and performance in accurately identifying and extracting bibliographic information from a wide range of academic documents.

3.2.1. Preprocessing Giant Corpus

For the purposes of this work, the annotated reference was simplified in an automated manner to maintain only the following labels, which are considered the minimum necessary to identify a work:

-Author	- Year	-Title	-Container-Title
-Volume	-Issue	-Page	-ISBN
-ISSN	-Publisher	-DOI	-URL

Resulting in the following format:

3.2.2. Preprocessing CORA Corpus

In the case of the CORA corpus, the labels were adjusted to align with those used in the Giant training, ensuring consistency across both datasets. The same preprocessing methodology applied to the Giant corpus was also employed for CORA, aiming to standardize the data and reduce variability and complexity. Additionally, 92 references were discarded from CORA due to encoding errors and label duplication, leaving 1787 of the original 1877 references for evaluation. Preprocessing ensures that both datasets are prepared to function correctly for training and evaluating the models, facilitating a direct and fair comparison of their performance on standardized bibliographic data.

4. Architectures of the Evaluated Models

4.1. CRF Model

The CRF model focuses on using Conditional Random Fields for sequence segmentation [22]. This technique is particularly effective in capturing dependencies and contextual patterns in sequential data (see Figure 1).

The following outlines the layers that comprise the architecture of the CRF model:

The description of the CRF model outlines its configuration and structure in terms of components and their functionalities:

Model: "CRF": This line denotes the name of the model, the following vignettes represent the layers that compose it:

• (embeddings): StackedEmbeddings: Refers to combining various types of word embeddings to form a rich and complex representation. It utilizes BytePairEmbeddings and CharacterEmbeddings:

  -

  (list_embedding_0): BytePairEmbeddings(model=0-bpe-multi-100000-50):BytePairEmbeddings are based on Byte Pair Encoding (BPE), capturing subword-level semantics, useful for handling out-of-vocabulary (OOV) words. The specific BPE model used is indicated by "model=0-bpe-multi-100000-50", detailing its parameters.

  -

  (list_embedding_1): CharacterEmbeddings: Uses character-level embeddings, crucial for understanding orthographic peculiarities and common errors in texts.

  *

  (char_embedding): Embedding(275, 25): Defines a character embedding with a vocabulary size of 275 and 25-dimensional vectors.

  *

  (char_rnn): LSTM(25, 25, bidirectional=True): A bidirectional LSTM network that processes character embeddings, with 25 units in both directions, capturing contexts before and after each character.

• (word_dropout): WordDropout(p=0.05): Applies dropout at the word level with a probability of 0.05, helping prevent overfitting by randomly "turning off" words during training.
• (locked_dropout): LockedDropout(p=0.5): Applies dropout uniformly across all dimensions of word vectors at a given step, with a probability of 0.5, enhancing the model’s robustness.
• (embedding2nn): Linear(in_features=650, out_features=650, bias=True): A linear layer transforming concatenated embeddings representation, preparing them for processing by subsequent layers.
• (linear): Linear(in_features=650, out_features=29, bias=True): Another linear layer acting as a classifier, mapping processed representations to 29 target label categories.
• (loss_function): ViterbiLoss(): Employs the Viterbi loss function, suitable for sequential prediction tasks like reference segmentation.
• (crf): CRF(): Indicates the use of a Conditional Random Field for sequence label prediction, optimizing the coherence and accuracy of predicted labels.

Each line succinctly summarizes a key component of the CRF model and its role in the learning and prediction process, highlighting the complexity and sophistication of the approach taken for bibliographic reference segmentation. Next, the equations describing the interactions of the components of the CRF model are presented.

embeddings:

$$\mathrm{emb}\left(w_i\right)=[\mathrm{emb}\mathrm{BPE}\left(w_i\right);\mathrm{emb}\mathrm{char}\left(w_i\right)]$$

(1)

The embeddings represent the combination of Byte Pair Encoding (BPE) and character embeddings. BPE captures the semantic aspects of words, while character embeddings focus on the syntactic nuances at the character level. This dual approach is crucial for processing bibliographic references, where both semantic context and specific syntactic forms (like abbreviations or special characters) play key roles.

word_dropout:

$${\mathrm{emb}}_{\mathrm{dropout}}\left(w_i\right)=\mathrm{Dropout}(\mathrm{emb}\left(w_i\right),p=0.05)$$

(2)

The word dropout randomly deactivates a portion of the word embeddings during training (here, 5% as indicated by p=0.05). This method prevents the model from over-relying on particular words, encouraging it to learn more generalized patterns. This approach is particularly beneficial in bibliographic texts where certain terms, such as common author names or publication titles, might appear with significantly higher frequency than others, potentially skewing the model’s learning focus.

locked_dropout:

$$\mathrm{emb}\mathrm{locked}\left(w_i\right)=\mathrm{LockedDropout}(\mathrm{emb}\mathrm{dropout}\left(w_i\right),p=0.5)$$

(3)

Locked dropout extends the dropout concept to entire embedding vectors, turning off the same set of neurons for the entire sequence. This approach helps in maintaining consistency in the representation of words across different contexts, an essential factor in processing structured bibliographic data.

embedding2nn:

$$\mathrm{emb}\mathrm{nn}\left(w_i\right)=\mathrm{Linear}(\mathrm{emb}\mathrm{locked}\left(w_i\right),650)$$

(4)

This linear transformation adapts the embeddings for further processing by the neural network layers. It is a crucial step for converting the rich, but potentially unwieldy, embedding information into a more suitable format for the classification tasks ahead.

linear:

$$\mathrm{output}\left(w_i\right)=\mathrm{Linear}({\mathrm{emb}}_{\mathrm{nn}}\left(w_i\right),29)$$

(5)

The final linear layer maps the transformed embeddings to the target classes. In this model, there are 29 classes, likely corresponding to different components of a bibliographic reference (like author, title, year, etc.). This layer is pivotal for the actual task of reference segmentation.

crf:

$$P\left(Y\right|H)=\frac{exp({\sum }_{i=1}^n{T}_{y_{i-1},y_i}+{\sum }_{i=1}^n{S}_{i,y_i})}{{\sum }_{Y^{\prime }}exp({\sum }_{i=1}^n{T}_{y^{\prime }i-1,y^{\prime }i}+\sum {i=1}^{n}Si,y_i^{\prime })}$$

(6)

$$Y^{*}=arg\underset{Y}{max}P\left(Y\right|H)$$

(7)

The CRF layer is key for capturing the dependencies between tags in the sequence. It considers not only the individual likelihood of each tag but also how likely they are in the context of neighboring tags. This sequential aspect is vital for bibliographic references, where the order and context of elements (like the sequence of authors or the structure of a citation) are crucial for accurate segmentation.

4.2. BiLSTM+CRF Model

The BiLSTM+CRF model combines Bidirectional Long Short-Term Memory (BiLSTM) networks with CRF [23] to better capturing both past and future context in the text sequence. BiLSTMs process the sequence in both directions, offering a deeper understanding of the syntactic structure (see Figure 2).

The following outlines the layers that comprise the architecture of the BiLSTM+CRF model:

This model is fundamentally similar to the CRF, with a key distinction being the incorporation of a Bidirectional Long Short-Term Memory ( highlighted in yellow as the rnn layer). This layer is strategically positioned between embedding2nn and the linear. The BiLSTM layer is crucial for capturing both past and future context, which is particularly beneficial for structured tasks like bibliographic reference segmentation.

The rnn layer is described below:

• (rnn): LSTM(650, 256, batch_first=True, bidirectional=True): A bidirectional LSTM layer that processes sequences in both forward and backward directions. With an input size of 650 features and an output of 256 features, it captures contextual information from both past and future data points within a sequence, enhancing the model’s ability to understand complex dependencies in bibliographic reference segmentation.

Let’s delve into the mathematical aspects of this layer:

rnn:

$$\overrightarrow{h_i}=\mathrm{BiLSTM}\mathrm{forward}(\mathrm{emb}\mathrm{nn}\left(w_i\right),\overrightarrow{h_{i-1}})$$

(8)

$$\overleftarrow{h_i}=\mathrm{BiLSTM}\mathrm{backward}(\mathrm{emb}\mathrm{nn}\left(w_i\right),\overleftarrow{h_{i+1}})$$

(9)

$$h_i=[\overrightarrow{h_i};\overleftarrow{h_i}]$$

(10)

These equations represent the forward and backward passes of the BiLSTM. The forward pass $\overrightarrow{h_i}$ processes the sequence from start to end, capturing the context up to the current word. Conversely, the backward pass $\overleftarrow{h_i}$ processes the sequence in reverse, capturing the context from the end to the current word. The final hidden state $h_i$ is a concatenation of these two passes, providing a comprehensive view of the context surrounding each word.

This bidirectional context is invaluable for bibliographic data. For instance, in a list of authors, the context of surrounding names helps in accurately identifying the start and end of each author’s name. Similarly, for titles or journal names, the BiLSTM can effectively use the context to delineate these components accurately.

4.3. Transformer+CRF Model

Finally, the model incorporating the Transformer Encoder [24] with CRF leverages the architecture of transformers for global attention processing of the sequence. This approach allows capturing complex and non-linear relationships in the data (see Figure 3).

The following outlines the layers that comprise the architecture of the Transformer+CRF model:

The positional_enconder, transformer_encoder_layer and transformer_encoder layers are described below:

• (positional_encoding): PositionalEncoding(dropout=0.1, inplace=False): This layer adds positional information to the input embeddings, allowing the model to capture the sequence of the text. The addition of positional encodings is crucial for attention-based models, such as transformers, as it enables them to distinguish the order of words in a sequence. The use of dropout with a probability of 0.1 helps to prevent overfitting by randomly "turning off" parts of the positional embeddings during training to enhance the model’s robustness.
• (transformer_encoder_layer): CustomTransformerEncoderLayer(...) and (transformer_encoder): TransformerEncoder(...): These two layers represent the heart of the Transformer model, where the first defines the structure of a single layer of the transformer encoder, including multi-head attention, residual connections, and layer normalization, while the second stacks multiple of these layers to construct the complete encoder. The apparent redundancy between these layers is because the first specifies the architecture and configuration of an individual layer within the encoder, including specific operations such as attention and normalization, and the second encapsulates the repetition of these layers to form the complete encoder, allowing the model to process and learn from sequences with greater depth and complexity. The inclusion of BatchNorm1d in the custom layer suggests an adaptation to stabilize and accelerate training by normalizing activations across mini-batches, which is not typical in standard transformers but can offer benefits in terms of convergence and performance in specific tasks like bibliographic reference segmentation.

It should be mentioned that in the case of the Transformer+CRF model, placing the Transformer Encoder layer before the embedding2nn layer is due to the following reasons.

First, in bibliographic references, context is of vital importance. The position of a word or phrase can significantly alter its interpretation, such as distinguishing between an author’s name and the title of a work. By processing the embeddings through the positional encoder and the Transformer from the beginning, the model can more effectively capture the contextual and structural relationships specific to bibliographic references.

Second, the early inclusion of the Transformer allows for early capture of these contextual relationships. This is crucial in bibliographic references, where the structure and order of elements (authors, title, publication year, etc.) follow patterns that can be complex and varied. The Transformer, known for its ability to handle long-distance dependencies, is ideal for detecting and learning these patterns.

Finally, once contextualized representations are generated, the embedding2nn layer acts as a fine-tuner, making specific adjustments and improvements to these representations. This makes the representations even more suitable for the Named Entity Recognition (NER) task, optimally adapting them for the accurate identification of the different components within bibliographic references.

The model being analyzed here, while structurally similar to the CRF model, introduces a significant variation with the addition of positional_encoding and a transformer_encoder_layer between the embedding and word_dropout layers. This inclusion is a key differentiator, enhancing the model’s ability to process and understand the sequence data more effectively. Let’s explore these additional layers:

positional_encoding

$${\mathrm{emb}}_{\mathrm{pos}}\left(w_i\right)=\mathrm{PositionalEncoding}\left(\mathrm{emb}\left(w_i\right)\right)$$

(11)

Positional encoding is added to the embeddings to provide context about the position of each word in the sequence. This is particularly crucial in transformer-based models, as they do not inherently capture sequential information. By incorporating positional data, the model can better understand the order and structure of elements in bibliographic references, such as distinguishing between the start and end of titles or authors’ lists.

transformer_encoder

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$$

(12)

$$\mathrm{transformer\_out}\left(w_i\right)=\mathrm{TransformerEncoderLayer}\left({\mathrm{emb}}_{\mathrm{pos}}\left(w_i\right)\right)$$

(13)

$$\begin{array}{cc}\hfill {\mathrm{X}}_{\mathrm{input}}& ={\mathrm{emb}}_{\mathrm{pos}}\left(w_i\right)\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill {\mathrm{X}}_{\mathrm{att}}& =\mathrm{MultiheadAttention}({\mathrm{X}}_{\mathrm{input}},{\mathrm{X}}_{\mathrm{input}},{\mathrm{X}}_{\mathrm{input}})\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill {\mathrm{X}}_{\mathrm{dropout}}& =\mathrm{Dropout}\left({\mathrm{X}}_{\mathrm{att}}\right)+{\mathrm{X}}_{\mathrm{input}}\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill {\mathrm{X}}_{\mathrm{norm}1}& =\mathrm{LayerNorm}\left({\mathrm{X}}_{\mathrm{dropout}}\right)\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill {\mathrm{X}}_{\mathrm{intermediate}}& =\mathrm{Linear}1\left({\mathrm{X}}_{\mathrm{norm}1}\right)\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill {\mathrm{X}}_{\mathrm{output}}& =\mathrm{Linear}2\left({\mathrm{X}}_{\mathrm{intermediate}}\right)\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill {\mathrm{X}}_{\mathrm{dropout}2}& =\mathrm{Dropout}\left({\mathrm{X}}_{\mathrm{output}}\right)+{\mathrm{X}}_{\mathrm{norm}1}\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill \mathrm{transformer\_out}\left(w_i\right)& =\mathrm{LayerNorm}\left({\mathrm{X}}_{\mathrm{dropout}2}\right)\hfill \end{array}$$

(21)

The transformer encoder layer employs multi-head attention mechanisms, enabling the model to focus on different parts of the sequence simultaneously. This multi-faceted approach is beneficial for bibliographic references, as it allows the model to capture complex relationships and dependencies between different parts of a reference, such as correlating authors with titles or publication years. The layer also includes several normalization and dropout steps, ensuring stable and efficient training.

Each of these models was carefully designed and optimized for reference segmentation, considering both accuracy in identifying reference components and computational efficiency.

5. Experiments and Results

6. Conclusions

References

1. Khabsa, M.; Giles, C.L. The number of scholarly documents on the public web. PloS one 2014, 9, e93949. [Google Scholar] [CrossRef]
2. Ware, M.; Mabe, M. The STM report: An overview of scientific and scholarly journal publishing 2015.
3. Bornmann, L.; Mutz, R. Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references. Journal of the association for information science and technology 2015, 66, 2215–2222. [Google Scholar] [CrossRef]
4. Becker, D.A.; Chiware, E.R. Citation Analysis of Master’s Theses and Doctoral Dissertations: Balancing Library Collections With Students’ Research Information Needs. Journal of Academic Librarianship 2015, 41, 613–620. [Google Scholar] [CrossRef]
5. Rizvi, S.T.R.; Dengel, A.; Ahmed, S. A Hybrid Approach and Unified Framework for Bibliographic Reference Extraction. IEEE Access 2020, 8, 217231–217245. [Google Scholar] [CrossRef]
6. Jain, V.; Baliyan, N.; Kumar, S. Machine Learning Approaches for Entity Extraction from Citation Strings. International Conference on Information Technology. Springer, 2023, pp. 287–297.
7. Choi, W.; Yoon, H.M.; Hyun, M.H.; Lee, H.J.; Seol, J.W.; Lee, K.D.; Yoon, Y.J.; Kong, H. Building an annotated corpus for automatic metadata extraction from multilingual journal article references. PLoS one 2023, 18, e0280637. [Google Scholar] [CrossRef] [PubMed]
8. Bergmark, D. Automatic extraction of reference linking information from onlinedocuments. Cornell University 2000.
9. Hetzner, E. A simple method for citation metadata extraction using hidden markov models. Proceedings of the 8th ACM/IEEE-CS joint conference on Digital libraries, 2008, pp. 280–284.
10. Patro, S.; Wang, W. Learning Top- k Transformation Rules. Database and Expert Systems Applications, 2011, pp. 172–186.
11. Peng, F.; McCallum, A. Information extraction from research papers using conditional random fields. Information Processing and Management 2006, 42, 963–979. [Google Scholar] [CrossRef]
12. Lopez, P. GROBID: Combining automatic bibliographic data recognition and term extraction for scholarship publications. International conference on theory and practice of digital libraries. Springer, 2009, pp. 473–474.
13. Councill, I.G.; Giles, C.L.; Kan, M.Y. ParsCit: An open-source CRF Reference String and Logical Document Structure Parsing Package. Proceedings of the 6th International Conference on Language Resources and Evaluation, 2008, Vol. 8, pp. 661–667.
14. Prasad, A.; Kaur, M.; Kan, M.Y. Neural ParsCit: a deep learning-based reference string parser. International Journal on Digital Libraries 2018, 19, 323–337. [Google Scholar] [CrossRef]
15. Rodrigues Alves, D.; Colavizza, G.; Kaplan, F. Deep Reference Mining From Scholarly Literature in the Arts and Humanities. Frontiers in Research Metrics and Analytics 2018, 3. [Google Scholar] [CrossRef]
16. Tkaczyk, D.; Gupta, R.; Cinti, R.; Beel, J. ParsRec: A novel meta-learning approach to recommending bibliographic reference parsers. CEUR Workshop Proceedings, 2018, Vol. 2259, pp. 162–173, [1811.10369].
17. Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 2018, arXiv:1810.04805. [Google Scholar]
18. Grennan, M.; Schibel, M.; Collins, A.; Beel, J. Giant: The 1-billion annotated synthetic bibliographic-reference-string dataset for deep citation parsing. CEUR Workshop Proceedings, 2019, Vol. 2563, pp. 260–271.
19. Anzaroot, S.; McCallum, A. A new dataset for fine-grained citation field extraction. ICML 2013 Workshop on Peer Reviewing and Publishing Models 2013.
20. Grennan, M.; Beel, J. Synthetic vs. Real Reference Strings for Citation Parsing, and the Importance of Re-training and Out-Of-Sample Data for Meaningful Evaluations: Experiments with GROBID, GIANT and Cora. arxiv.org 2020, pp. 1–7, [2004.10410].
21. Akbik, A.; Bergmann, T.; Blythe, D.; Rasul, K.; Schweter, S.; Vollgraf, R. FLAIR: An easy-to-use framework for state-of-the-art NLP. NAACL 2019, 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), 2019, pp. 54–59.
22. Lafferty, J.; McCallum, A.; Pereira, F.C. Conditional random fields: Probabilistic models for segmenting and labeling sequence data 2001.
23. Huang, Z.; Xu, W.; Yu, K. Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint 2015, arXiv:1508.01991. [Google Scholar]
24. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser; Polosukhin, I. Attention is all you need. Advances in neural information processing systems 2017, 30. [Google Scholar]
25. Yan, H.; Deng, B.; Li, X.; Qiu, X. TENER: adapting transformer encoder for named entity recognition. arXiv preprint 2019, arXiv:1911.04474. [Google Scholar]
26. Mitrofan, M.; Păiș, V. Improving Romanian BioNER using a biologically inspired system. Proceedings of the 21st Workshop on Biomedical Language Processing, 2022, pp. 316–322.