Preprint Article Version 1 This version is not peer-reviewed

Modification and Validation of the System Causability Scale Using AI-Based Therapeutic Recommendations for Urological Cancer Patients: A Basis for the Development of a Prospective Comparative Study

Emily Rinderknecht
,
Dominik von Winning
,
Anton Kravchuk
,
Christof Schäfer
,
Marco J. Schnabel
,
Stephan Siepmann
,
Roman Mayr
,
Jochen Grassinger
,
Christopher Goßler
,
Fabian Pöhl
,
Peter J. Siska
,
Florian Zeman
,
Johannes Breyer
,
Anna Magdalena Schmelzer
,
Christian Gilfrich
,
Sabine. D. Brookman-May
,
Maximilian Burger
,
Maximilian Haas
,
Matthias May
*
Version 1 : Received: 16 October 2024 / Approved: 16 October 2024 / Online: 16 October 2024 (16:26:36 CEST)

How to cite: Rinderknecht, E.; Winning, D. V.; Kravchuk, A.; Schäfer, C.; Schnabel, M. J.; Siepmann, S.; Mayr, R.; Grassinger, J.; Goßler, C.; Pöhl, F.; Siska, P. J.; Zeman, F.; Breyer, J.; Schmelzer, A. M.; Gilfrich, C.; Brookman-May, S. D.; Burger, M.; Haas, M.; May, M. Modification and Validation of the System Causability Scale Using AI-Based Therapeutic Recommendations for Urological Cancer Patients: A Basis for the Development of a Prospective Comparative Study. Preprints 2024, 2024101290. https://doi.org/10.20944/preprints202410.1290.v1 Rinderknecht, E.; Winning, D. V.; Kravchuk, A.; Schäfer, C.; Schnabel, M. J.; Siepmann, S.; Mayr, R.; Grassinger, J.; Goßler, C.; Pöhl, F.; Siska, P. J.; Zeman, F.; Breyer, J.; Schmelzer, A. M.; Gilfrich, C.; Brookman-May, S. D.; Burger, M.; Haas, M.; May, M. Modification and Validation of the System Causability Scale Using AI-Based Therapeutic Recommendations for Urological Cancer Patients: A Basis for the Development of a Prospective Comparative Study. Preprints 2024, 2024101290. https://doi.org/10.20944/preprints202410.1290.v1

Abstract

The integration of artificial intelligence, particularly Large Language Models (LLMs), has the potential to significantly enhance therapeutic decision-making in clinical oncology. Initial studies across various disciplines have demonstrated that LLM based treatment recommendations can rival those of multidisciplinary tumor boards (MTBs); however, such data are currently lacking for urological cancers. This study provides the methodological foundation for the prospective CONCORDIA trial, which will generate this data for the first time. In this preliminary work, we evaluated the proposed measurement tool for the CONCORDIA study -the System-Causability-Scale (SCS) and its modified version (mSCS) - based on recommendations from ChatGPT-4 and an MTB for 40 urological cancer-scenarios. Both scales demonstrated strong validity, reliability (all aggregated Cohen’s K > 0.74), and internal consistency (all Cronbach’s Alpha > 0.9), with the mSCS showing superior reliability, internal consistency, and clinical applicability (p<0.01). Two Delphi processes were used to define the LLMs to be tested in the CONCORDIA study (ChatGPT-4 and Claude 3.5 Sonnet) and to establish the acceptable non-inferiority margin for LLM recommendations compared to MTB recommendations. The forthcoming ethics-approved and registered CONCORDIA non-inferiority trial will require 110 urological cancer scenarios, with an mSCS difference threshold of 0.15, a Bonferroni corrected alpha of 0.025, and a beta of 0.1. Blinded mSCS assessments of MTB recommendations will then be compared to those of the LLMs. In summary, this work establishes the necessary prerequisites prior to initiating the CONCORDIA study and validates a modified score with high applicability and reliability for this and future trials.

Keywords

Artificial Intelligence Integration; Large Language Models; Multidisciplinary Tumor Boards; Non-inferiority CONCORDIA Trial; SCS; Urological Cancer Treatment; Validation Study; Clinical Decision Support; Artificial Neural Network

Subject

Medicine and Pharmacology, Oncology and Oncogenics

Copyright: This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Download PDF

Comments (0)

We encourage comments and feedback from a broad range of readers. See criteria for comments and our Diversity statement.

Leave a public comment
Send a private comment to the author(s)
* All users must log in before leaving a comment
Views 0
Downloads 0
Comments 0
Get PDF Cite Share 0
Bookmark BibSonomy Mendeley Reddit Delicious
×
Alerts
Notify me about updates to this article or when a peer-reviewed version is published.
We use cookies on our website to ensure you get the best experience.
Read more about our cookies here.