Version 1
: Received: 2 May 2024 / Approved: 2 May 2024 / Online: 2 May 2024 (10:14:36 CEST)
How to cite:
Nechakhin, V.; D’Souza, J.; Eger, S. Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph. Preprints2024, 2024050124. https://doi.org/10.20944/preprints202405.0124.v1
Nechakhin, V.; D’Souza, J.; Eger, S. Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph. Preprints 2024, 2024050124. https://doi.org/10.20944/preprints202405.0124.v1
Nechakhin, V.; D’Souza, J.; Eger, S. Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph. Preprints2024, 2024050124. https://doi.org/10.20944/preprints202405.0124.v1
APA Style
Nechakhin, V., D’Souza, J., & Eger, S. (2024). Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph. Preprints. https://doi.org/10.20944/preprints202405.0124.v1
Chicago/Turabian Style
Nechakhin, V., Jennifer D’Souza and Steffen Eger. 2024 "Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph" Preprints. https://doi.org/10.20944/preprints202405.0124.v1
Abstract
Structured science summaries or research contributions using properties or dimensions beyond traditional keywords enhances science findability. Current methods, such as those used by the Open Research Knowledge Graph (ORKG), involve manually curating properties to describe research papers' contributions in a structured manner, but this is labor-intensive and inconsistent between the domain expert human curators. We propose using Large Language Models (LLMs) to automatically suggest these properties. However, it's essential to assess the readiness of LLMs like GPT-3.5, Llama 2, and Mistral for this task before application. Our study performs a comprehensive comparative analysis between ORKG's manually curated properties and those generated by the aforementioned state-of-the-art LLMs. We evaluate LLM performance through four unique perspectives: semantic alignment and deviation with ORKG properties, fine-grained properties mapping accuracy, SciNCL embeddings-based cosine similarity, and expert surveys comparing manual annotations with LLM outputs. These evaluations occur within a multidisciplinary science setting. Overall, LLMs show potential as recommendation systems for structuring science, but further finetuning is recommended to improve their alignment with scientific tasks and mimicry of human expertise.
Keywords
Large Language Models; Open Research Knowledge Graph; Structured Summarization
Subject
Computer Science and Mathematics, Information Systems
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
