Preprint Article Version 1 Preserved in Portico This version is not peer-reviewed

An Improved Evaluation Metrics for Sentence Suggestions in Nursing and Elderly Care Record Applications

Version 1 : Received: 15 December 2023 / Approved: 15 December 2023 / Online: 18 December 2023 (06:02:58 CET)

A peer-reviewed article of this Preprint also exists.

Hamdhana, D.; Kaneko, H.; Victorino, J.N.; Inoue, S. Improved Evaluation Metrics for Sentence Suggestions in Nursing and Elderly Care Record Applications. Healthcare 2024, 12, 367. Hamdhana, D.; Kaneko, H.; Victorino, J.N.; Inoue, S. Improved Evaluation Metrics for Sentence Suggestions in Nursing and Elderly Care Record Applications. Healthcare 2024, 12, 367.

Abstract

In this paper, we propose a novel approach named EmbedHDP to enhance the evaluation models used to assess sentence suggestions within nursing care record applications. The key focus is determining whether these suggestions garner assessments that align with caregivers as human evaluators. It is crucial due to the direct relevance of the information provided to the health or condition of the elderly. The motivation behind this proposal stems from challenges observed in previous models, such as BERTScore, which encountered difficulties in effectively evaluating the nurse care record domain, consistently providing quality assessments of generated sentence suggestions above 60%. Additionally, while widely used, cosine similarity exhibits limitations concerning word order, leading to potential misjudgments of semantical differences within similar word sets. Similarly, relying on lexical overlap, ROUGE tends to overlook semantic accuracy. Furthermore, despite its utility, BLEU neglects semantic coherence in its evaluations. EmbedHDP excels in evaluating nurse care records by effectively handling a variety of sentence structures and medical terminology and providing differentiated and contextually relevant assessments. We used a dataset comprising 320 pairs of sentences with correspondingly equivalent lengths. The results revealed that EmbedHDP outperformed other evaluation models, achieving a coefficient score of 61%, followed by cosine similarity with a score of 59%, and BERTScore with 58%. This shows the effectiveness of our proposed approach in improving the evaluation of sentence suggestions in nursing care record applications.

Keywords

sentence suggestion; nursing care record; evaluation metrics; elderly care record

Subject

Public Health and Healthcare, Nursing

Comments (0)

We encourage comments and feedback from a broad range of readers. See criteria for comments and our Diversity statement.

Leave a public comment
Send a private comment to the author(s)
* All users must log in before leaving a comment
Views 0
Downloads 0
Comments 0


×
Alerts
Notify me about updates to this article or when a peer-reviewed version is published.
We use cookies on our website to ensure you get the best experience.
Read more about our cookies here.