Version 1
: Received: 21 August 2024 / Approved: 22 August 2024 / Online: 22 August 2024 (08:25:51 CEST)
How to cite:
Nakamura, K.; Tatsuoka, H.; Miyakawa, T.; Ishii, H.; Onishi, M.; Ohyama, Y. Evaluation and Prototype of Health Counseling LLM Using the Turing Test. Preprints2024, 2024081614. https://doi.org/10.20944/preprints202408.1614.v1
Nakamura, K.; Tatsuoka, H.; Miyakawa, T.; Ishii, H.; Onishi, M.; Ohyama, Y. Evaluation and Prototype of Health Counseling LLM Using the Turing Test. Preprints 2024, 2024081614. https://doi.org/10.20944/preprints202408.1614.v1
Nakamura, K.; Tatsuoka, H.; Miyakawa, T.; Ishii, H.; Onishi, M.; Ohyama, Y. Evaluation and Prototype of Health Counseling LLM Using the Turing Test. Preprints2024, 2024081614. https://doi.org/10.20944/preprints202408.1614.v1
APA Style
Nakamura, K., Tatsuoka, H., Miyakawa, T., Ishii, H., Onishi, M., & Ohyama, Y. (2024). Evaluation and Prototype of Health Counseling LLM Using the Turing Test. Preprints. https://doi.org/10.20944/preprints202408.1614.v1
Chicago/Turabian Style
Nakamura, K., Masahiro Onishi and Yoshiaki Ohyama. 2024 "Evaluation and Prototype of Health Counseling LLM Using the Turing Test" Preprints. https://doi.org/10.20944/preprints202408.1614.v1
Abstract
In 2021, Japan's medical expenses will exceed 45 trillion yen, and the shortage of doctors, especially in remote and mountainous areas, is becoming serious, making it difficult to maintain the medical system. We have conducted a study of 800 health consultation text data. We developed an on-premise health counseling LLM system by constructing a dialogue flow based on 800 health counseling text data. We conducted a Turing test of this system using 200 test data and verified its effectiveness with three medical professionals. The Turing test was a comparison experiment between this system and a conventional LLM system. The health counseling LLM infrastructure focuses on exercise guidance and analyzes gender, height, weight, body fat percentage, and muscle mass. While the accuracy of the conventional LLM system was 87.5%, this system showed a higher accuracy of 93.1%. Although telemedicine has been slow to spread in Japan, the introduction of a health consultation system using Personal Health Record and Large Language Models has the potential to reduce the burden on physicians. In the future, we aim to improve the accuracy of the system by using Japanese language and medical-specific evaluation scales.
Keywords
Telemedicine; Health Counseling; Large Language Models (LLM); Turing Test; Health counseling; Artificial Intelligence in Healthcare; Remote Health Monitoring; Personal Health Record (PHR)
Subject
Engineering, Bioengineering
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
