1. Introduction

2. Materials and Methods

3. Results

3.1. About the Conceptual Model

The resulting conceptual model is presented in Figure 1. The domain of clinical effectiveness is related to the accurate screening of dengue, the re-consultation rate, and the severity of the patient condition, classified as: recovered at home; recovered after being hospitalized; immediately referred to another hospital with a higher level of complexity; recovered in another hospital.

Figure 1. Causal diagram of the POCUS assessment model with all four assessment domains and LC.

Figure 1. Causal diagram of the POCUS assessment model with all four assessment domains and LC.

The security domain is related to the rate of false negatives and false positives associated with dengue screening.

The organizational domain refers to the average time between the performance of the ultrasound and the training course. Infrastructure-related aspects were not considered in this domain since they do not affect the portability of the device.

In terms of the economic domain, an analysis of fixed and variable costs and consequences was carried out to form a value judgment on the use of technology with information on related costs.

The feedback loops (Figure 1) that represent the dynamics of the subsystems are described below:

-

Loop B1 (Balance): It represents the relationship between costs and indicators for decision making. When decision-making criteria such as cost-benefit ratio, availability, false negatives and false positives, and adequate screening of extravasation are increased, the associated costs decrease. In cases where the rigor of the mentioned processes decreases, the costs increase. These parameters can be adjusted by the decision-maker through the Stella interface.

-

Loop B2 (Balance): By using the POCUS equipment for the diagnosis of diseases other than dengue, variable costs decrease due to the reimbursement received by insurers for the provision of these services. As variable costs increase, total costs increase. Finally, when the total costs increase, it means that the reimbursement for the service of other diagnoses decreases.

Loop R1 (Reinforcement): It shows the causal relationship between the LC and the indicators for decision making; the behavior of the curve in terms of a better learning curve will represent a better cost-benefit ratio.

In relation to the LC the Monte Carlo simulation, it was concluded:

For LC-CUSUM the acceptable and unacceptable failure rates were: P₁=0,1 y P₀= 0,3, with a limit H: 2,5 and for CUSUM the acceptable and unacceptable failure rates were: P₀=0,1 and P₁= 0,2, with a limit H: 1,5. These values comply with the Average Run Length.

The performance of LC-CUSUM in terms of the Average Run Length (ARL₀) was 98 procedures under the null hypothesis and 17 procedures under the alternative hypothesis (ARL₁), 17 procedures. For CUSUM the ARL₀ was 99 procedures and ARL₁ was 23 procedures on average. The LC design parameters were taken from an article approved for publication in an indexed journal. [40]

For each sequential image (t), the sequential score is calculated as ${\mathrm{S}}_{\mathrm{t}}=\max \left(0,{\mathrm{S}}_{\mathrm{t}-1}+{\mathrm{W}}_{\mathrm{t}}\right)$ and Wt is determined based on the next equation:

$$W_t=\left\{\begin{array}{c}\mathit{Log}\left[\left(1-P_1\right)/\left(1-P_0\right)\right]\mathit{if}{X}_{t=}0(\mathit{success})\\ \mathit{Log}\left[P_1/P_o\right]\mathit{if}{X}_{t=}1(\mathit{failure})\end{array}\right\}$$

In this work, the sequence S_t was modeled as a Bernoulli distribution (simulated in STELLA® as a binomial distribution with N = 1) with an 80% probability of success. It is possible to adjust this parameter in the simulation.

3.2. About the Case Study

3.2.1. Patient demand module

This module is related to the demand from patients that could have dengue, are confirmed to have dengue, are re-entry patients, are recovering at home, are hospitalized, and are sent to other healthcare institutions.

The demand from patients that could have dengue was simulated through the Poisson distribution of the case study data in days throughout a year. The model determined that confirmed cases of dengue corresponded to patients with positive IgM with a rate of 0.56 (100 / 178).

Furthermore, a possible second evaluation was considered when the person still has dengue symptoms and reenters the model after being labeled as a false negative. Although the case study did not include readmission data, the simulation allows the user to set this parameter. A rate of 0.29 (52/178) was programmed into the model using the consultations taking place during the checkup window as a reference.

The Wilcoxon signed-rank test was applied, resulting in p = 0.476, which implies that there is no significant difference between simulated and real data (Figure 2).

Figure 2. Patient intake behavior. The model follows the behavior of the case study and exhibits an increase in intake for the second semester of the year.

Figure 2. Patient intake behavior. The model follows the behavior of the case study and exhibits an increase in intake for the second semester of the year.

The following equations represent the behavior of the model: D_c are the patients confirmed to have dengue; O_d are the patients with other diagnoses; P_HyR are the patients that recovered in the hospital and were referred; P_H patients that are hospitalized in the institution; P_R patients that were referred immediately; and P_Rd patients that were referred after a few days. Considering the cohort study and the routine checkup of patients, the hospitalization rate was set at 0.67 (67/100), the immediate referral rate at 0.02 (2/100), and the rate of being referred after a few days at 0.13 (9/67).

$$P_{withsymptom}(t)=P_{withsymptom}(t_o)+{\int }_0^t(P_{withsymptom}-D_c-O_d)dt$$

$$P_{HyR}(t)=P_{HyR}(t_o)+{\int }_0^t(P_{HyR}+P_H+P_R+P_{Rd})dt$$

3.2.2. POCUS healthcare module

The number of patients with and without plasma leakage is calculated in this module using a rate of 0.83 (149/178) according to the cohort study. It is assumed that all patients suspected of dengue take the POCUS test. However, if the patient has pulmonary compromise and the doctor is not qualified to perform the POCUS test, then the patient is referred to the radiology area for a chest x-ray. The rate for chest x-rays was defined as 0.511 (91/178) which corresponds to patients with positive IgM and leakage confirmed by the expert radiologist.

The true positive and negative values VP and VN and the false positive and negative values FP and FN were calculated according to the equations detailed below. The sensitivity VPN and specificity VPP calculations were based on [41]. However, the user can modify the parameters of the model.

$$\mathrm{V}\mathrm{P}\_calculated=\frac{\mathrm{P}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\mathrm{l}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{a}\mathrm{g}\mathrm{e}\ast \mathrm{S}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{i}\mathrm{t}\mathrm{y}}{100}$$

$$\mathrm{V}\mathrm{N}\_calculated=\frac{\mathrm{P}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{l}\mathrm{e}\mathrm{a}\mathrm{k}\mathrm{a}\mathrm{g}\mathrm{e}\ast \mathrm{S}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}}{100}$$

$${\mathrm{F}\mathrm{P}}_{\_calculated}=\frac{{\mathrm{V}\mathrm{N}}_{\_calculated}}{VP{P}_{\_literature}}-{\mathrm{V}\mathrm{P}}_{\_calculated}$$

$${\mathrm{F}\mathrm{N}}_{\_calculated}=\frac{{\mathrm{V}\mathrm{N}}_{\_calculated}}{VP{N}_{\_literature}}-{\mathrm{V}\mathrm{N}}_{\_calculated}$$

The model delivered a total of 159 patients with plasma leakage, which represents a 6% error compared to real data. According to simulations, if the doctor were trained enough from day 1 and never lost said skillset, the POCUS test would be performed for dengue screening on 177 patients. Given the effect of the learning curve and the base scenario of 20% failure probability, 131 patients entered the emergency room when the doctor was untrained to carry out the POCUS test, and 44 of them would take a chest x-ray. Therefore, the POCUS test is performed on 46 patients (177-131). In Figure 3, when the red dotted signal “reached proficiency status” reaches the value of 1, the patients are not sent to the radiology department, and the insurance company is charged for the POCUS test.

Figure 3. The red dotted signal "reached proficiency status " is at 1, indicating that patients will not be sent to the radiology department.

Figure 3. The red dotted signal "reached proficiency status " is at 1, indicating that patients will not be sent to the radiology department.

3.2.3. Ponderation and training module

As a result of the Montecarlo simulation and based on the equation presented in section 3.1., this module computed W(t) for LC-CUSUM in success cases, obtaining 0.2513, and in failure cases, obtaining -0.0986. The W(t) for CUSUM was -0.1178 in success cases and 0.6931 in failure cases.

The learning curve obtained in the base scenario of an 80% success rate for the doctor is presented in Figure 4 (solid blue line). The training level is reached on day 85, and this level is maintained until day 257; then the training level is recovered on day 290, lost on day 306, and never recovered. The red dotted line represents the number of patients that could have dengue when entering the emergency room.

Figure 4. learning curve for base scenario of 80% success rate.

Figure 4. learning curve for base scenario of 80% success rate.

Figure 5 depicts the moments in which the training must be repeated in a black line, which are days 258 and 306.

Figure 5. Behavior of the learning curve and retraining for base scenario of 80% success rate.

Figure 5. Behavior of the learning curve and retraining for base scenario of 80% success rate.

3.2.4. POCUS-Dengue cost module

The fixed costs are computed in this module related to purchase value of the training biomedical equipment, usage of the equipment and annual preventive maintenance.

The reference values are taken from the Colombian market. The model assumes an equipment cost of COP $25.000.000, based on the Lumify Phillips device with a curve matrix of 5 to 2 MHz in B mode, and an equipment cost for emergency care of COP $101.000.000, based on the Sonosite Edge 2 device with these transductors: linear from 13 to 6 MHz, curved from 8 to 3 MHz, and endocavitary from 5 to 8 MHZ, in modes B, M, and color. The reference lifecycle of the equipment is 9 years, after which new echographs need to be purchased. The simulation took into consideration a preventive maintenance cost equal to 4% of the purchase price.

The fixed costs included training for POCUS-Dengue, using as a reference the certification in emergency and critical care medicine, with a timeframe of 90 hours, aimed at medical specialists (anesthesiologists, intensivists, surgeons, ER doctors, internists, etc.) in charge of handling critical care patients. The topics encompass the physical principles of ultrasound and instrumentation, basic focused echocardiography (FOCUS), echography of the lungs and pleura, FAST, aortic echography, echography for Deep Vein Thrombosis (DVT), and ultrasound-guided vascular access.

The costs associated with the POCUS-Dengue procedure included the retraining of doctors who lost their skillset. This information is generated by the model in the training module. Another factor is the workforce needed to cover for the clinical staff while they are in training. The model considers that at least three general doctors should be trained, one for each ER shift. This parameter can be adjusted by the user.

The variable costs were related to the supplies needed for echography such as gel and wipes, as well as the retraining process when the doctor needs to repeat the POCUS-dengue certification over time.

In terms of the income received by the healthcare institution from the insurance company due to the POCUS service, the fee corresponds to the one presented in the Fee Manual for Colombia. This income is only factored under the premise that the doctor is trained to perform the test on 46 patients.

The costs and projection of income were determined throughout the equipment lifecycle.

The dengue patient demand involved an analysis of the track record of Colombia. Between 2014 and 2022, it was found that 2019 was an epidemiological year, and there was no increased tendency afterwards. Hence, the model contemplates demand throughout the product lifecycle.

A 5% projection of costs and income was estimated, using as a reference the salary increase in Colombia between 2017 and 2021. The return on investment (ROI) was determined, as were the net present value and the cost-benefit ratio. A social discount of 9% was applied, as established by the National Department of Planning for the assessment of investment projects in the public sector.

In the base scenario, the cost-benefit ratio is 0.15, meaning that costs exceed benefits, and the ROI is negative for every year of the simulation, marking the lifecycle of the POCUS equipment.

The validation of the model was based on the coherence of units and extreme condition analysis. When a model is exposed to extreme conditions, the researchers can verify whether the model offers reasonable results and, thus, is error-free in its structure. The model showed consistent behavior in every module. One example of this is the demand behavior and Wilcoxon signed-rank test, with no significant difference between simulated and real data linked to the case study.

4. Discussion

5. Conclusions

References

1. World Health Organization. Health Intervention and Technology Assessment in Support of Universal Health Coverage.; 2014.
2. Danish National Board of Health. Introduction to Mini-HTA- a Management and Decision Support Tool for the Hospital Service. (Danish Centre for Evaluation and Health Technology Assessment (DACEHTA), ed.).; 2005. doi:10.1049/sm.1982.0029. [CrossRef]
3. Kristensen FB, Lampe K, Wild C, Cerbo M, Goettsch W, Becla L. The HTA Core Model s — 10 Years of Developing an International Framework to Share Multidimensional Value Assessment. Value Heal. 2017;20(2):244-250. doi:10.1016/j.jval.2016.12.010. [CrossRef]
4. Sampietro-Colom L LK, Cicchetti A KK, Pasternack I FB, Rosenmöller M WC, Kahveci R WJ, Kiivet RA et al. The AdHopHTA Handbook: A Handbook of Hospital-based Health Technology Assessment (HB-HTA).; 2015. http://www.adhophta.eu/sites/files/adhophta/media/adhophta_handbook_website.pdf.
5. Tarricone R, Torbica A, Drummond M. Key Recommendations from the MedtecHTA Project. Heal Econ (United Kingdom). 2017;26:145-152. doi:10.1002/hec.3468. [CrossRef]
6. Fuchs S, Olberg B, Panteli D, Busse R. Health Technology Assessment of Medical Devices in Europe: Processes, Practices, and Methods. Int J Technol Assess Health Care. 2016;32(04):246-255. doi:10.1017/S0266462316000349. [CrossRef]
7. Fuchs S, Olberg B, Panteli D, Perleth M, Busse R. HTA of medical devices: Challenges and ideas for the future from a European perspective. Health Policy (New York). 2017;121(3):215-229. doi:10.1016/j.healthpol.2016.08.010. [CrossRef]
8. Tarricone R, Torbica A, Drummond M. Challenges in the Assessment of Medical Devices: The MedtecHTA Project. Heal Econ (United Kingdom). 2017;26:5-12. doi:10.1002/hec.3469. [CrossRef]
9. Soneja S, Tsarouchi G, Lumbroso D, Tung DK. A Review of Dengue’s Historical and Future Health Risk from a Changing Climate. Curr Environ Heal Reports. 2021;8(3):245-265. doi:10.1007/s40572-021-00322-8. [CrossRef]
10. World Health Organisation. Dengue and Severe Dengue. Vol 117.; 2014. www.who.int/mediacentre/factsheets/fs117/en/index.html.
11. Pothapregada S, Kullu P, Kamalakannan B, Thulasingam M. Is Ultrasound a Useful Tool to Predict Severe Dengue Infection? Indian J Pediatr. 2016;83(6):500-504. doi:10.1007/s12098-015-2013-y. [CrossRef]
12. Bharath Kumar Reddy KR, Laksmana RR, Veerappa BG, Shivananda. Ultrasonography as a tool in predicting the severity of dengue fever in children - A useful aid in a developing country. Pediatr Radiol. 2013;43(8):971-977. doi:10.1007/s00247-013-2642-0. [CrossRef]
13. Bélard S, Joekes E, Tamarozzi F, et al. Point-of-Care Ultrasound Assessment of Tropical Infectious Diseases—A Review of Applications and Perspectives. Am J Trop Med Hyg. 2015;94(1):8-21. doi:10.4269/ajtmh.15-0421. [CrossRef]
14. Khurram M, Qayyum W, Umar M, Jawad M, Mumtaz S, Khaar HTB. Ultrasonographic pattern of plasma leak in dengue haemorrhagic fever. J Pak Med Assoc. 2016;66(3):260-264.
15. Vedaraju KS, Kumar KR V, Vijayaraghavachari T V. Role of Ultrasound in the Assessment of Dengue Fever. Int J Sci Study. 2016;3(10):59-62. doi:10.17354/ijss/2016/12. [CrossRef]
16. Brunetti E, Heller T, Richter J, et al. Application of Ultrasonography in the Diagnosis of Infectious Diseases in Resource-Limited Settings. Curr Infect Dis Rep. 2016;18(2):1-11. doi:10.1007/s11908-015-0512-7. [CrossRef]
17. Díaz-Gómez JL, Mayo PH, Koenig SJ. Point-of-Care Ultrasonography. N Engl J Med. 2021;385(17):1593-1602. doi:10.1056/NEJMra1916062. [CrossRef]
18. Biau DJ, Williams SM, Schlup MM, Nizard RS, Porcher R. Quantitative and individualized assessment of the learning curve using LC-CUSUM. Br J Surg. 2008;95(7):925-929. doi:10.1002/bjs.6056. [CrossRef]
19. Woodall WH, Rakovich G, Steiner SH. An overview and critique of the use of cumulative sum methods with surgical learning curve data. Stat Med. 2020;(November):1-14. doi:10.1002/sim.8847. [CrossRef]
20. Aminullah E, Erman E. Policy innovation and emergence of innovative health technology: The system dynamics modelling of early COVID-19 handling in Indonesia. Technol Soc. 2021;66. doi:10.1016/j.techsoc.2021.101682. [CrossRef]
21. Cassidy R, Tomoaia-Cotisel A, Semwanga AR, et al. Understanding the maternal and child health system response to payment for performance in Tanzania using a causal loop diagram approach. Soc Sci Med. 2021;285(July). doi:10.1016/j.socscimed.2021.114277. [CrossRef]
22. Mecoli M, De Angelis V, Brailsford SC. Using system dynamics to evaluate control strategies for mosquito-borne diseases spread by human travel. Comput Oper Res. 2013;40(9):2219-2228. doi:10.1016/j.cor.2012.03.007. [CrossRef]
23. Jayasinghe S, Zhu YG. Response to the commentary by M.W.C. Dharma-wardana on ‘Chronic kidney disease of unknown etiology (CKDu): Using a system dynamics model to conceptualize the multiple environmental causative pathways of the epidemic.’ Sci Total Environ. 2020;721:2019-2021. doi:10.1016/j.scitotenv.2020.137591. [CrossRef]
24. Sterman J. Business Dynamics : Systems Thinking and Modeling for a Complex World. Irwin/McGraw-Hill; 2000.
25. Davahli MR, Karwowski W, Taiar R. A system dynamics simulation applied to healthcare: A systematic review. Int J Environ Res Public Health. 2020;17(16):1-27. doi:10.3390/ijerph17165741. [CrossRef]
26. Barlas Y. Formal aspects of model validity and validation in system dynamics. Syst Dyn Rev. 1996;12(3):183-210. doi:10.1002/(SICI)1099-1727(199623)12:3<183::AID-SDR103>3.3.CO;2-W. [CrossRef]
27. Šilhánková V, Maštálka M. Urban Dynamics. Vol 27.; 2013. doi:10.4324/9781315438689-23. [CrossRef]
28. Lane DC, Sterman JD. Profiles in Operations Research: Jay Wright Forrester. In: International Series in Operations Research and Management Science. Vol 147. Springer New York LLC; 2011:363-386. doi:10.1007/978-1-4419-6281-2_20. [CrossRef]
29. Poder TG, Bellemare CA, Bédard SK, Fisette JF, Dagenais P. Impact Of Health Technology Assessment Reports On Hospital Decision Makers - 10-Year Insight From A Hospital Unit In Sherbrooke, Canada: Impact Of Health Technology Assessment On Hospital Decisions. Int J Technol Assess Health Care. 2018;34(4):388-392. doi:10.1017/S0266462318000405. [CrossRef]
30. Angelis A, Kanavos P. Value-Based Assessment of New Medical Technologies: Towards a Robust Methodological Framework for the Application of Multiple Criteria Decision Analysis in the Context of Health Technology Assessment. Pharmacoeconomics. 2016;34(5):435-446. doi:10.1007/s40273-015-0370-z. [CrossRef]
31. EUNETHA. Process of information retrieval for systematic reviews and health technology assessments on clinical effectiveness. 2016;(December). http://www.eunethta.eu/sites/default/files/Guideline_Information_Retrieval_V1-1.pdf.
32. Kidholm K, Ølholm AM, Birk-Olsen M, et al. Hospital managers’ need for information in decision-making - An interview study in nine European countries. Health Policy (New York). 2015;119(11):1424-1432. doi:10.1016/j.healthpol.2015.08.011. [CrossRef]
33. Martelli N, Hansen P, van den Brink H, et al. Combining multi-criteria decision analysis and mini-health technology assessment: A funding decision-support tool for medical devices in a university hospital setting. J Biomed Inform. 2016;59:201-208. doi:10.1016/j.jbi.2015.12.002. [CrossRef]
34. Ritrovato M, Faggiano FC, Tedesco G, Andellini M, Derrico P. Integrating AHP into EUNETHTA core model: the decision-oriented health technology assessment (doHTA) method. In: ; 2016. doi:10.13033/isahp.y2016.076. [CrossRef]
35. Ritrovato M, Faggiano FC, Tedesco G, Derrico P. Decision-oriented health technology assessment: One step forward in supporting the decision-making process in hospitals. Value Heal. 2015;18(4):505-511. doi:10.1016/j.jval.2015.02.002. [CrossRef]
36. Biau DJ, Porcher R. A method for monitoring a process from an out of control to an in control state: Application to the learning curve. Stat Med. 2010;29(18):1900-1909. doi:10.1002/sim.3947. [CrossRef]
37. Papanna R, Biau DJ, Mann LK, Johnson A, Moise KJ. Use of the Learning CurveCumulative Summation test for quantitative and individualized assessment of competency of a surgical procedure in obstetrics and gynecology: Fetoscopic laser ablation as a model. Am J Obstet Gynecol. 2011;204(3). doi:10.1016/j.ajog.2010.10.910. [CrossRef]
38. Arzola C, Carvalho JCA, Cubillos J, Ye XY, Perlas A. Anesthesiologists’ learning curves for bedside qualitative ultrasound assessment of gastric content: A cohort study. Can J Anesth. 2013;60(8):771-779. doi:10.1007/s12630-013-9974-y. [CrossRef]
39. Oliveira KF, Arzola C, Ye XY, Clivatti J, Siddiqui N, You-Ten KE. Determining the amount of training needed for competency of anesthesia trainees in ultrasonographic identification of the cricothyroid membrane. BMC Anesthesiol. 2017;17(1):1-7. doi:10.1186/s12871-017-0366-7. [CrossRef]
40. Usaquén-Perilla, S.P. Ropero-Rojas, D. Mosquera-Restrepo, J. D-Kirsh, J. P-Kaltenborn Z. García- Melo, J.I. Osorio L. Control charts to establish and monitor proficiency in the detection of pulmonary B- lines with Point of Care Ultrasound. Ing y Univ Eng Dev. 2023;(accepted).
41. Balasubramanian S, Janakiraman L, Shiv Kumar S, Muralinath S, Shivbalan S. A reappraisal of the criteria to diagnose plasma leakage in dengue hemorrhagic fever. Indian Pediatr. 2006;43(4):334-339.
42. Ramsay CR, Wallace SA, Garthwaite PH, Monk AF, Russell IT, Grant AM. Assessing the learning curve effect in health technologies. Lessons from the nonclinical literature. Int J Technol Assess Health Care. 2002;18(1):1-10. http://www.ncbi.nlm.nih.gov/pubmed/11987432.
43. Miniati R, Frosini F, Cecconi G, Dori F, Gentili GB. Development of sustainable models for technology evaluation in hospital. Technol Heal Care. 2014;22(5):729-739. doi:10.3233/THC-140847. [CrossRef]
44. Tolsgaard MG, Todsen T, Sorensen JL, et al. International Multispecialty Consensus on How to Evaluate Ultrasound Competence: A Delphi Consensus Survey. PLoS One. 2013;8(2). doi:10.1371/journal.pone.0057687. [CrossRef]
45. Schuwirth LWT, Van Der Vleuten CPM. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach. 2011;33(6):478-485. doi:10.3109/0142159X.2011.565828. [CrossRef]
46. RCR R college of R. Ultrasound Training Recommendations for Medical and Surgical Specialties Second Edition Board of the Faculty of Clinical Radiology.; 2015.