Preprint
Article
Can AI Technologies Support Automating Clinical Supervision? Assessing the Potential of ChatGPT
Altmetrics
Downloads
102
Views
55
Comments
0
This version is not peer-reviewed
Submitted:
23 October 2024
Posted:
24 October 2024
You are already at the latest version
Alerts
Abstract
Clinical supervision is an essential element in trainees, preventing burnout and ensuring the effectiveness of their intervention. AI technologies offer increasing possibilities for the development of clinical practices, among it supervision appears to possess the ideal characteristics for automation processes.
In this study we test the capabilities of ChatGPT-4 to provide supervisory feedback by then placing them in comparison with feedback produced by a qualified supervisor.
Two ChatGPT-4-generated feedbacks (the first one produced by a naïve identity and the second one by a trained identity) and one human-produced feedback were evaluated by means of a liking questionnaire, which was filled out to a group of gestalt psychotherapy trainees.
Principal Component Analysis (PCA) highlighted 4 components of the questionnaire: relational and emotional dimensions (C1), didactic and technical quality (C2), treatment support and development (C3), and professional orientation and adaptability (C4). The ratings of satisfaction, obtained from the three supervisory feedbacks, were compared by applying 1-way analysis of variance (ANOVA). Statistical evaluations were performed using the statistical package SPSS version 25.
The feedback generated by pre-trained AI (f2) was rated significantly higher than the other two (untrained AI feedback (f1) and human feedback (f3) in C4; in C1 the superiority of f2 over f1 but not over f3 appears significant.
These results suggest that the use of pre-trained AI may be an appreciable option for increasing the effectiveness of clinical supervision in some specific areas, including especially career guidance.
Keywords:
Subject: Public Health and Healthcare - Health Policy and Services
Introduction
Clinical supervision is a crucial element in psychotherapeutic treatments. It consists of a meeting between a psychotherapist and a colleague, more expert and trained for the purpose, aimed at providing support for clinical case management through the presentation of another point of view that is external to the dynamics of the therapist-patient dyad. It is an indispensable element in the professional development of therapists, both in its just-described form of clinical supervision, in which it is the technical work with a specific patient that is being supervised, and in its declination of professional supervision where, to be assisted is, in a more direct way the professional growth of the therapist. Supervision is essential for the professional development of therapists, both with a view to preventing burn-out and enhancing their clinical effectiveness [1].
According to Watkins Jr, C.E. (2020) “Psychotherapy supervisors serve first and foremost as agents of transformation, their main goal being to implement and actualize a transformative process of therapist development.” [2]
As highlighted by Yontef, G. (1996), colleagues in training, such as trainees and interns, through supervision can enhance awareness of their current therapeutic skills, thus laying an indispensable foundation to further enhance them [3].
One of the most notorious problems inherent in the practice of the health professions and, in particular, those related to mental health, is the emotional depletion of the therapist. The resulting risk of burn out can be managed or prevented through supervision, which, turns out to be a key element for this purpose. It also serves as a catalyst in the personal and professional development of psychotherapists, especially in the early stages of their careers [4]. Indeed, the well-being of the therapist is an indispensable factor for the quality of clinical intervention in terms of efficiency and effectiveness [5]. The new potential for clinical practice offered by technological development is made evident by several studies [6,7,8,9]. In this context, we place clinical supervision in relation to artificial intelligence technologies of the “Generative Pre-Trained Transformer” type, in this case ChatGPT-4, as a new angle of investigation to delve into how they can be used and their possible usefulness for the improvement and implementation of clinical supervision, in order to verify more and more accurately the possibility of making them a complementary tool for the pursuit of the purposes of classical supervision. [10]
Aims
This paper is part of a broad program aimed at exploring the potential of using ChatGPT as a work support for psychotherapists by testing the AI’s ability to provide useful feedback for conducting psychotherapy sessions. In the specifics of this study, we evaluated psychotherapists-in-training’s satisfaction of supervisory feedback produced by a naïve ChatGPT-4 identity, from one of its identities trained according to a model described in previous work [10].
The ultimate purpose of the curriculum in which this paper is embedded is to identify ways in which ChatGPT can be used more to support the work the work of clinicians in what concerns the therapeutic process and the professional development of therapists, with a view to complementing traditional supervision settings.
Methodology
We submitted a clinical case (anonymized) to two different ChatGPT-4 interfaces; the text provided AI-chatbot included medical history data, question analysis, a previously conducted psychodiagnostic assessment, a detailed description of the patient’s personality functioning, and a description, from the therapist’s perspective, of the first three sessions of psychotherapy, with a special focus on the therapist’s subjective experience in session.
We proposed the clinical case described above to two different chat interfaces: in one (test 1), we presented the case without pretraining the chat session regarding the scope of expertise of gestalt therapy and related supervision; in a further chat interface (test 2), we introduced the scope of expertise using a prompt as a pretraining tool, before proceeding with the presentation of the case. This prompt was generated using ChatGPT and its reliability as a pre-training tool for the chat session was verified as a good alternative to a longer and more complex training process in a previous study we conducted [10]
In parallel (test 3), the same document presented to the chatbot was studied by an expert psychotherapist and supervisor who wrote their own feedback. The first feedback places a specific focus on both the strengths of the therapeutic process and the areas in need of improvement, providing, in addition, an evaluation of the therapeutic strategies used. The second feedback, on the other hand, obtained through the use of the prompt, places more emphasis on the therapist’s use of “self,” that is, how he or she uses his or her personal experiences in therapy, emphasizing the importance of supervision in managing transference and countertransference. Finally, the third contribution focuses more on the therapeutic relationship through two reflections that provide a better understanding of countertransference dynamics and the use of metaphors in therapy.
The two AI-generated feedbacks, along with the one written by an experienced supervisor were brought to the attention of a group of trainees in integrated gestalt psychotherapy, after a comprehensive clinical case presentation and description of the sessions. The presentation of the 4 feedback items to the evaluators was blind, with no information given as to how and/or by whom they were written.
For each of the 4 items, they were asked to provide an evaluation by responding to a likert questionnaire based on a 4-grade likert scale (1= Poor/Fair; 2=Sufficient/Little; 3=Good/Fair; 4=Excellent/Very Good). The development of the above questionnaire (Table 1) was partially inspired by P. Clarkson’s model of supervision in Transactional Analysis. According to the author, supervision constitutes a continuous and hierarchical process, characterized by the figure of a directive and responsible supervisor and a supervisee who is required to report regularly on the status of his or her work [11]. The seven components characterizing this model of supervision are: 1. clear and appropriate contract; 2. key problems identified; 3. effective emotional contact with the trainee; 4. protection of both trainee and patient; 5. increased developmental directions; 6. awareness and effective use of the parallel process; and 7. equal relationship. The author discusses the characteristics of these seven components in relation to the three different stages of trainee development (beginning, intermediate, advanced) [11].
Results
For the purpose of data analysis, the 3 feedbacks evaluated were assigned the following designations: the one generated by ChatGPT without prior interactions was named “Feedback 1” or “Fb1”; the one generated by ChatGPT after pre-training by means of a special prompt [10] was named “Feedback 2” or “Fb2”; the Feedback written by a supervising qualified psychotherapist was named “Feedback 3.”
While the first feedback places a specific focus both on the strengths of the therapeutic process and on the areas that need improvement, the second feedback highlights more the use of the therapist’s "self", i.e. the way in which he uses their personal experiences in therapy, underlining the importance of supervision for the management of transference and countertransference. Finally, the third contribution focuses on the therapeutic relationship through two reflections that allow a better understanding of the countertransference dynamics and the use of metaphors in therapy.
The proposed 16-item questionnaire for each feedback was completed by 25 subjects for Fb1, by 24 for Fb2, and by 22 for Fb3, for a total of 71 ratings. Sufficient to conduct a principal component analysis (Table 2).
Through this analysis, 4 components were identified that group the areas investigated by the 16 items.
PCA, using varimax rotation method, identified the following dimensions:
- RELATIONAL AND EMOTIONAL DIMENSION includes: empathic approach, usefulness of reflection, confidence support, emotional impact (on the evaluator).
- DIDACTIC AND TECHNICAL QUALITY includes: clarity, relevance, thoroughness, analysis of techniques.
- TREATMENT SUPPORT AND DEVELOPMENT includes: how helpful the feedback is to the treatment; presence of practical suggestions to the therapist; how much it highlights areas for improvement; how equal communication appears.
- PROFESSIONAL ORIENTATION AND ADAPTABILITY includes: how deontologically oriented the feedback is; how much the feedback helps the contract; how well the feedback fits the therapist’s professional level; how helpful the feedback is to the therapist’s professional development. The individual component scores were obtained by summing the likert scale values adjusted for the value of the coefficients. PCA explains 68.179% of the variance.
A one-way analysis of variance was then conducted to assess the differences between the three feedbacks for each identified component (Table 3). This analysis revealed a significant internal difference between the feedback ratings for component 1 (relational and emotional dimensions) and component 4 (career orientation and adaptability); while such significance is not found for the other two components.
In the subsequent analysis of multiple comparisons (Table 4), the significance of the differences in the ratings received to the 3 feedbacks for the individual components was assessed.
Regarding component one, the differences in ratings between fb 1 and fb 2 and between fb 1 and fb 3 were significant; in short, fb 1 appears to differ from both other fb. No significant difference is observed between fb 2 and fb 3.
As for components two and three, on the other hand, there is no significant difference in the feedback comparisons.
While regarding component four there is a significant difference between fb 1 the fb 2, but not between fb 1 and fb3. Also, a significant difference between fb 2 and fb 3 is observed, unlike what can be observed in component one.
To make a specific comparison between the AI-generated feedback (Fb1 and Fb2) and the human feedback (Fb3), a comparison was made between the ratings to the individual items, regardless of the components previously identified (Table 4). A statistically significant difference emerged between the 3 feedbacks in the items related to empathic approach, contract help, appropriateness to the supervisee’s professional level, highlighting areas for improvement, trust help, and emotional impact. Feedback 2, generated by the pre-trained AI, is the one that received a higher average score than the untrained AI in the items related to: empathetic approach, help with the contract, suitability for the professional level, highlighting areas for improvement and helps with trust. In the item related to emotional impact, however, the pre-trained AI (fb 2) received a higher average score than both the untrained AI (fb 1) and human feedback (fb3).
Discussions
This study explored the effectiveness of CHAT-GPT-4 in the area of psychotherapeutic supervision: this was done through comparison of feedback generated by trained and untrained AI with that of an experienced human supervisor. The results highlight a potential effectiveness of AI, trained through the use of a specific prompt, in providing accurate supervisory feedback in specific areas.
As noted in the results, 4 components were identified from the principal component analysis that grouped the different areas investigated by the 16 items designed to assess the quality of the 3 different supervisory feedbacks. Specifically, the analysis of variance showed that there were differences in the ratings of the three feedback items with respect to the first component “Relational and Emotional Dimension” and the fourth component “Professional Orientation.” With the subsequent analysis of multiple comparisons, it was possible to investigate how the ratings of the three feedbacks differed with respect to the above two components. The results showed:
(a) A higher satisfaction with fb2 and fb3 than with fb1, regarding the relational and emotional dimension; (b) A higher satisfaction with fb2 than with both fb1 and fb3, regarding the dimension related to professional orientation.
The first result would seem to indicate a greater effectiveness of pretrained AI (fb2) than non- pretrained AI (fb1) in providing supervisory feedback with respect to the relational and emotional dimensions, which relate to: the empathic approach, the usefulness of reflection, helping trust, and the emotional impact on the evaluator.
This result seems to be linked to the AI’s pre-training conducted through the prompt, which enabled it to focus on relational and emotional aspects, core elements of Gestalt psychotherapy. [10]
The second result, on the other hand, would attest to a greater effectiveness of the pretrained AI (fb 2) compared to both the non-pretrained AI (fb1) and the supervising experienced psychotherapist (fb3) in providing supervisory feedback with respect to the dimension of professional orientation, which concerns aspects related to: how deontologically oriented the feedback is, how much it helps the contract, how well it fits the therapist’s professional level, and, finally, how useful it is for the therapist’s professional development.
Therefore, the second feedback, focusing more on the therapist’s use of the “self” and on the importance of supervision for the management of transference and countertransference, is more effective in framing the dimension of professional orientation, if compared to the first and to the third contribution. In fact, the first one places a specific focus both on the strengths of the therapeutic process and on the areas that need improvement; however, the third, similarly to the feedback generated by the pre-trained AI, mostly focuses on the therapeutic relationship and the subjectivity of the therapist as an element of the field [12], highlighting a better understanding of the countertransference dynamics and the use of metaphors in therapy.
This result seems to highlight, the possibility of using AI to enrich the training process of therapists and for accurate and thorough information management and data analysis. [14]
In contrast, the subsequent specific comparison of AI-generated feedback (fb1 and fb2) and human feedback (fb3), carried out by comparing ratings to individual items, without considering the previously identified components, revealed: a) A higher mean score of fb2 with respect to the individual items of emotional approach, contract help, appropriateness to professional level, confidence help, and emotional impact; b) A higher mean score of fb3 with respect to the item on highlighting areas for improvement. Specifically, the first result would indicate the greater effectiveness of the pre-trained AI (fb2) compared to both the non-pre-trained AI (fb1) and the expert therapist supervisor (fb3) in providing supervisory feedback in terms of emotional approach, assistance with the therapeutic contract, appropriateness to the professional level, support in building confidence, and emotional impact. This result could reflect AI’s ability to analyze and process large amounts of data and use this function to formulate responses that resonate linguistically empathetic or knowledgeable with respect to the situations described [13]
The second result, on the other hand, would highlight the greater effectiveness of the supervising expert psychotherapist (fb3) compared to the AI (fb1 and fb2) in capturing and highlighting areas in need of improvement.
A higher average score of fb2 than fb1 for the individual items of the emotional approach, help with the contract, suitability for the professional level, highlighting areas for improvement and help with trust.
A higher average score of fb2 than both fb1 and fb3 for the item relating to the emotional impact. In particular, the first result would attest to the greater effectiveness of the pre-trained AI (fb2) compared to the non-pre-trained AI (fb1) in providing supervisory feedback with respect to the empathic approach, help with the contract, suitability for the professional level, highlighting areas for improvement and help with trust, obtaining results like those of human feedback. This result could reflect the ability of the AI to analyze and process large amounts of data and to use this function to formulate responses that sound linguistically empathetic or aware with respect to the situations described [13].
The second result, instead, would highlight the greater effectiveness of the pre-trained AI (fb2) compared to both the non-pre-trained AI (fb1) and the expert psychotherapist supervisor (fb3) in providing feedback focused on the emotional impact.
This result is also in line with a similar study involving 600 participants to evaluate the level of empathy in the responses generated by humans and ChatGPT: this research also highlighted that the average empathy rating of the responses generated by ChatGPT exceeded those provided by humans by about 10% to a variety of positive and negative emotional situations [14].
Conclusions
This paper contributed to give a demonstration of how the ways in which AI can be used to make it a complementary tool to support psychotherapists’ clinical work and training. It became evident, in fact, how supervisory feedback provided by an adequately pre-trained interface of CHAT-GPT- 4, was perceived by psychotherapists-in-training as fully valid, and, in part, comparable to human feedback.
In research, it is interest gaining ground for the potential uses of Artificial Intelligence models in psychotherapy, also with promising results that highlight it as a valid complementary tool to clinical practice [15]. However, the literature does not yet seem to highlight more specific areas of interest regarding particular aspects of psychotherapists’ work, such as supervision and specialist training. With this paper, in continuity with the previous one [10], we also intended to fill this gap, proposing to the scientific and professional community this more targeted area of investigation. Psychotherapy, as both a science and an art focused on the relationship with the human being, recognizing their complexity, authenticity, and individuality in an inseparable connection with the ever-changing world, remains alive over time by evolving, updating itself, and adopting new forms in continuous co- adaptation with the transformations in the human-society-culture field [16]. The integration of new languages, new forms and “places” (physical and non-physical) of interaction and relationship [17] and the new possibilities offered by technology must necessarily be part of this integration. As already outlined in the introduction to this work, clinical and professional supervision in the psychotherapeutic field assumes, among other things, an irreplaceable tool for preventing burnout [5]. However, the realization of a supervision meeting is not always immediately feasible when the need arises: integrating a valid AI-based tool as a support tool for the therapist, which he can use between supervision meetings, can prove extremely useful in the quality of his management of his own well-being and his own effectiveness in clinical practice. As discussed below, in future developments of this work, such integration could occur by cyclically making: 1) the reports of the supervision sessions ChatGPT training material by the therapist; 2) the supervision feedback provided by the AI object of real supervision. The Long-Term Memory function, recently implemented in ChatGPT-4, would make this process easily achievable since, differently from what happened when the foundations of this study were laid, the learnings of said AI within the same account are used transversally in all conversation interfaces.
Limitations And Future Developments
The present study has some limitations related to the methodology, in that it is a single-case study, and to the sample, which, due to the low numerosity and homogeneity of the subjects, could limit the generalizability of the results obtained. The sample selected for this study, consisting only of psychotherapy trainees, in fact, would require the involvement of psychotherapists with different levels of professional development.
The failure to evaluate the diverse sociocultural backgrounds of the participants, which could have influenced their evaluations of the feedback, may also represent a limitation of this research. Many studies, in fact, have highlighted that the perception of empathy is different in various cultures [18,19]. A further limitation of the present research may be the significant ethical issues it raises: although AI can provide technically correct supervisory feedback, its ability to understand and respond to human emotional complexities remains limited. As such, its application should be carefully balanced with human input in contexts requiring emotional sensitivity and ethics such as psychotherapy.
As already mentioned in the conclusions of this work, we could think that the integration of the strengths of both types of feedback, the one generated by AI and the human one, could result in a blended supervision methodology, characterized by the possibility of integrating a first supervisory feedback generated by AI with a second level supervision, carried out by an expert supervisor starting from the contribution of the AI itself. This integration of human creativity and AI systematicity, in a perspective of mutual support and learning, would allow for greater timeliness and accuracy of the supervision intervention, also preventing phenomena particularly frequent in helping professions such as burnout.
Ultimately, this study, highlighting the effectiveness of AI in providing feedback assessed as empathically valid, could suggest the possibility in the future of implementing a tool based on AI itself, which through the monitoring of the subjects’ involuntary physiological responses, such as heart rate, breathing and muscle tone, uses these data to develop the ability to provide outputs that have a more empathic impact [20,21].
References
- Giusti, E., Montanari, C., Spalletta, E. (2000). La supervisione clinica integrata. Manuale di formazione pluralistica in counseling e psicoterapia. Italia: Elsevier.
- Watkins Jr, C. E. (2020). The psychotherapy supervisor as an agent of transformation: To anchor and educate, facilitate and emancipate. American Journal of Psychotherapy, 73(2), 57-62. [CrossRef]
- Yontef, G. (1996). Supervision from a Gestalt therapy perspective. British Gestalt Journal, 5, 92-102. [CrossRef]
- Rønnestad, M. H., Orlinsky, D. E., Schröder, T. A., Skovholt, T. M., & Willutzki, U. (2019). The professional development of counsellors and psychotherapists: Implications of empirical studies for supervision, training and practice. Counselling and Psychotherapy Research, 19(3), 214-230. [CrossRef]
- Watkins Jr, C. E. (2011). Does psychotherapy supervision contribute to patient outcomes? Considering thirty years of research. The clinical supervisor, 30(2), 235-256. [CrossRef]
- Cioffi, V., Mosca, L. L., Moretto, E., Ragozzino, O., Stanzione, R., Bottone, M., ... & Sperandeo, R. (2022). Computational Methods in Psychotherapy: A Scoping Review. International Journal of Environmental Research and Public Health, 19(19), 12358. [CrossRef]
- Tahan, M., & Zygoulis, P. (2020). Artificial Intelligence and Clinical Psychology, Current Trends. Journal of Clinical & Developmental Psychology, 2(1). [CrossRef]
- Miner, A. S., Shah, N., Bullock, K. D., Arnow, B. A., Bailenson, J., & Hancock, J. (2019). Key considerations for incorporating conversational AI in psychotherapy. Frontiers in psychiatry, 10, 746. [CrossRef]
- Luxton, D. D. (2014). Artificial intelligence in psychological practice: Current and future applications and implications. Professional Psychology: Research and Practice, 45(5), 332. [CrossRef]
- Cioffi, V., Ragozzino, O., Scognamiglio, C., Mosca, L. L., Moretto, E., Stanzione, R., Marino, F., Acocella, A., Ammendola, A., D’Aquino, R., Durante, S., Tortora, E., Morfini, F., Montanari, C., Rosa, V., Rossi, O., Ferrara, A., Mori, E., Gigante, E., Pizzimenti, M., Zangarini, S., Sperandeo R. & Cantone D.(2024). Towards integrated AI psychotherapy supervision: A proposal for a ChatGPT-4 study. In press.
- Clarkson,P., Psicoterapia analitico-transazionale: un approccio integrato, Routledge, Londra 1992.
- Lewin, K. (1951). Teoria del campo delle scienze sociali. Selected Theorical Papers, Hardcover.
- Manzotti, R., & Rossi, S. (2023). IO & IA Mente, Cervello & GPT. Rubbettino.
- Welivita, A., &; Pu, P. (2024). Is ChatGPT More Empathetic than Humans?. arXiv preprint arXiv:2403.05572.
- Raile, P. (2024). The usefulness of ChatGPT for psychotherapists and patients. Humanities and Social Sciences Communications, 11(1), 1-8. [CrossRef]
- Francesetti, G., Gecele, M., & Roubal, J. (Eds.). (2014). La psicoterapia della Gestalt nella pratica clinica. Dalla psicopatologia all’estetica del contatto: Dalla psicopatologia all’estetica del contatto. FrancoAngeli.
- Lancini, M. (2020). Il ritiro sociale negli adolescenti: la solitudine di una generazione iperconnessa. Raffaello Cortina Editore.
- Melissa Birkett. 2014. Autocompassione ed empatia Attraverso le culture: confronto tra i giovani adulti in Cina e gli Stati Uniti. Giornale Internazionale di Re cerca Studi in psicologia, 3(3):25–34.
- William J Chopik, Ed O’Brien e Sara H Konrath. 2017. Differenze nella preoccupazione empatica e nella prospettiva tive in 63 paesi. Giornale della Croce Psicologia culturale, 48(1):23–38.
- Blackmore, K. L., Smith, S. P., Bailey, J. D., & Krynski, B. (2024). Integrating Biofeedback and Artificial Intelligence into eXtended Reality Training Scenarios: A Systematic Literature Review. Simulation & Gaming, 55(3), 445-478. [CrossRef]
- Sperandeo, R., Di Sarno, A. D., Longobardi, T., Iennaco, D., Mosca, L. L., & Maldonato, N. M. (2019). Toward a technological oriented assessment in psychology: a proposal for the use of contactless devices for heart rate variability and facial emotion recognition in psychological diagnosis. In PSYCHOBIT.
Table 1.
Satisfaction questionnaire.
| ITEMS | Not at All/Poor | A Little/ Sufficient | Quite /Good |
Very Good/ Excellent |
|---|---|---|---|---|
| Did you find the feedback clear? | 1 | 2 | 3 | 4 |
| Is this feedback relevant to the clinical case presented? | 1 | 2 | 3 | 4 |
| Is this feedback comprehensive? | 1 | 2 | 3 | 4 |
| Did this feedback help the supervisee define the therapeutic contract with the patient? | 1 | 2 | 3 | 4 |
| Does this feedback provide useful insights for the patient’s treatment? | 1 | 2 | 3 | 4 |
| Does this feedback take into account the supervisee’s professional level? | 1 | 2 | 3 | 4 |
| Does this feedback contribute to the professional development of the therapist? | 1 | 2 | 3 | 4 |
| Did this feedback adequately address the ethical and deontological aspects of the clinical case? | 1 | 2 | 3 | 4 |
| Does this feedback provide practical suggestions useful for the therapist? | 1 | 2 | 3 | 4 |
| Does this feedback offer an analysis of the techniques used in the sessions? | 1 | 2 | 3 | 4 |
| Does this feedback constructively highlight any areas for improvement? | 1 | 2 | 3 | 4 |
| Did reading this feedback have an emotional impact on you? | 1 | 2 | 3 | 4 |
| Do you consider this feedback to be characterized by an empathetic approach? | 1 | 2 | 3 | 4 |
| Does this feedback come across as a collegial communication between peers? | 1 | 2 | 3 | 4 |
| Did this feedback help strengthen the supervisee’s capacity for self-reflection? | 1 | 2 | 3 | 4 |
| Has this feedback improved the supervisee’s confidence in clinical case management? | 1 | 2 | 3 | 4 |
Table 2.
Principal component analysis (PCA) extracted from the 16-item liking test.
| Component | ||||
|---|---|---|---|---|
| C1: Relational and Emotional Dimension |
C2: Didactic and Technical Quality |
C3: Treatment Support and Development |
C4: Professional Orientation and Adaptability | |
| empathic approach | 0.812 | |||
| It helps self-reflection | 0.633 | |||
| It helps confidence | 0.741 | |||
| emotional impact | 0.776 | |||
| clarity | 0.846 | |||
| relevance | 0.544 | |||
| completeness | 0.733 | |||
| analysis of techniques | 0.637 | |||
| treatment aids | 0.489 | |||
| practical suggestions to the therapist | 0.82 | |||
| It highlights areas for improvement | 0.806 | |||
| peer-to-peer communication | 0.512 | |||
| deontologically oriented | 0.578 | |||
| It helps contract | 0.667 | |||
| appropriate to professional level | 0.772 | |||
| useful for professional development |
0.683 | |||
| Extraction method: Principal component analysis. | ||||
| Rotation method: Varimax with Kaiser normalization. the Convergence for rotation performed in 7 iterations. | ||||
| KMO measure of sampling adequacy = 0.860 | ||||
Table 3.
ANOVA for comparing the ratings of the 4 components across the three feedbacks.
| Average | Standard Deviation | of 95% Confidence Interval | |||||
|---|---|---|---|---|---|---|---|
| Lower Limit | Upper Limit | F | p | ||||
| C1: Relational and emotional dimension | fb1 | 10.395 | 2.773 | 9.250 | 11.540 | 5.733 | 0.005 |
| fb2 | 13.011** | 2.255 | 12.058 | 13.962 | |||
| fb3 | 11.982 | 3.114 | 10.601 | 13.363 | |||
| C2: Didactic and Technical Quality | fb1 | 12.502 | 2.370 | 11.524 | 13.481 | 1.287 | 0.283 |
| fb2 | 13.625 | 2.327 | 12.642 | 14.607 | |||
| fb3 | 12.849 | 2.802 | 11.606 | 14.091 | |||
| C3: Treatment support and development | fb1 | 11.281 | 2.059 | 10.431 | 12.131 | 0.269 | 0.765 |
| fb2 | 10.966 | 2.153 | 10.057 | 11.875 | |||
| fb3 | 11.450 | 2.631 | 10.283 | 12.616 | |||
| C4: Professional Orientation and Adaptability | fb1 | 10.423 | 2.080 | 9.564 | 11.282 | 3.385 | 0.04 |
| fb2 | 11.911* | 1.961 | 11.082 | 12.739 | |||
| fb3 | 10.438 | 2.74 | 9.223 | 11.652 | |||
fb1: untrained AI feedback - fb2: trained AI feedback - fb3: human feedback. **significantly different score with both two scores. significantly different score between fb2 and fb3.
Table 4.
ANOVA for comparing the ratings of the 16 items across the three feedbacks.
| Average | Standard Deviation | of 95% confidence interval for average | |||||
|---|---|---|---|---|---|---|---|
| Items | LL | UL | F | p | |||
| clarity | fb1 | 3.08 | 0.64 | 2.82 | 3.34 | 2.42 | 0.097 |
| fb2 | 3.33 | 0.702 | 3.04 | 3.63 | |||
| fb3 | 2.86 | 0.834 | 2.49 | 3.23 | |||
| relevance | fb1 | 3.08 | 0.702 | 2.79 | 3.37 | 0.887 | 0.417 |
| fb2 | 3.29 | 0.69 | 3 | 3.58 | |||
| fb3 | 3.32 | 0.646 | 3.03 | 3.6 | |||
| completeness | fb1 | 2.88 | 0.781 | 2.56 | 3.2 | 0.914 | 0.406 |
| fb2 | 3.13 | 0.612 | 2.87 | 3.38 | |||
| fb3 | 2.86 | 0.834 | 2.49 | 3.23 | |||
| empathic approach | fb1 | 2.28 | 0.843 | 1.93 | 2.63 | 17.408 | >0,001 |
| fb2 | 3.33* | 0.637 | 3.06 | 3.6 | |||
| fb3 | 2.64 | 1.002 | 2.19 | 3.08 | |||
| deontologically oriented | fb1 | 2.68 | 0.852 | 2.33 | 3.03 | 0.212 | 0.647 |
| fb2 | 2.63 | 0.824 | 2.28 | 2.97 | |||
| fb3 | 2.36 | 1.002 | 1.92 | 2.81 | |||
| It helps the contract | fb1 | 2.28 | 0.678 | 2 | 2.56 | 7.062 | 0.01 |
| fb2 | 2.88* | 0.797 | 2.54 | 3.21 | |||
| fb3 | 2.41 | 0.908 | 2.01 | 2.81 | |||
| It hepls the treatment | fb1 | 2.88 | 0.6 | 2.63 | 3.13 | 0.067 | 0.797 |
| fb2 | 2.96 | 0.751 | 2.64 | 3.28 | |||
| fb3 | 3.14 | 0.941 | 2.72 | 3.55 | |||
| analysis of techniques | fb1 | 2.84 | 0.8 | 2.51 | 3.17 | 0.164 | 0.687 |
| fb2 | 2.75 | 0.737 | 2.44 | 3.06 | |||
| fb3 | 2.82 | 0.795 | 2.47 | 3.17 | |||
| practical suggestions to the therapist | fb1 | 2.96 | 0.841 | 2.61 | 3.31 | 1.465 | 0.23 |
| fb2 | 2.83 | 0.761 | 2.51 | 3.15 | |||
| fb3 | 3.18 | 0.733 | 2.86 | 3.51 | |||
| suitable for professional level | fb1 | 2.32 | 0.852 | 1.97 | 2.67 | 13.325 | 0.001 |
| fb2 | 3.17* | 0.868 | 2.8 | 3.53 | |||
| fb3 | 2.41 | 0.908 | 2.01 | 2.81 | |||
| useful for professional development | fb1 | 3 | 0.645 | 2.73 | 3.27 | 2.241 | 0.139 |
| fb2 | 3.29 | 0.751 | 2.97 | 3.61 | |||
| fb3 | 3 | 0.926 | 2.59 | 3.41 | |||
| It highlights areas for improvement | fb1 | 3.28 | 0.678 | 3 | 3.56 | 5.634 | 0.02 |
| fb2 | 2.88 | 0.741 | 2.56 | 3.19 | |||
| fb3 | 3.32* | 0.716 | 3 | 3.64 | |||
| It helps self-reflection | fb1 | 2.72 | 0.614 | 2.47 | 2.97 | 1.409 | 0.239 |
| fb2 | 3.17 | 0.702 | 2.87 | 3.46 | |||
| fb3 | 3.18 | 0.853 | 2.8 | 3.56 | |||
| It helps confidence | fb1 | 2.44 | 0.821 | 2.1 | 2.78 | 7.648 | 0.007 |
| fb2 | 3.17* | 0.637 | 2.9 | 3.44 | |||
| fb3 | 2.82 | 0.853 | 2.44 | 3.2 | |||
| emotional impact | fb1 | 1.88 | 0.833 | 1.54 | 2.22 | 6.378 | 0.014 |
| fb2 | 2.96** | 0.908 | 2.57 | 3.34 | |||
| fb3 | 2.86 | 1.037 | 2.4 | 3.32 | |||
| peer communication | fb1 | 2.52 | 0.77 | 2.2 | 2.84 | 0.523 | 0.472 |
| fb2 | 2.38 | 0.875 | 2.01 | 2.74 | |||
| fb3 | 2.55 | 0.963 | 2.12 | 2.97 | |||
fb1: untrained AI feedback, fb2: trained AI feedback, fb3: human feedback. **significantly different score with both two scores. *significantly different score between fb1 and fb2.
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
MDPI Initiatives
Important Links
© 2024 MDPI (Basel, Switzerland) unless otherwise stated