Preprint
Brief Report
Planning Audit and Feedback Interventions in Health Care Organisations. An Account from an Italian National Program for Audit & Feedback Implementation
Altmetrics
Downloads
99
Views
41
Comments
0
This version is not peer-reviewed
Submitted:
03 July 2023
Posted:
06 July 2023
You are already at the latest version
Alerts
Abstract
Audit&Feedback (A&F) consists of multidimensional quality improvement activities, but its optimal design is still debated. Our study aims to outline how interventions of the Italian EASY-NET research program were designed, to explore whether desirable characteristics of an "ideal" A&F were adopted. A&F design included description of the working group, targeted clinical behaviors, selected indicators, sources, feedback procedures, corrective actions expected. Information was classified into four topics: nature of the desired action, type of data available, feedback display and delivery. Out of 9 projects, 3/9 aimed at changing a focused clinical behavior, 6/9 were generic; all projects identified clinicians as recipients, 8 explicitly considered managers. It was planned to develop an average of 27 indicators from administrative databases, as needed supplemented by ad hoc data collection. Five projects included outcome measures. Comparators derived from scientific literature or from territorial real-life data. Feedback was scheduled every 12, 6 or 3 months, or upon request, delivered as aggregate data (graphs and tables) via web platforms, e-mail, workshops, and individual meetings. In conclusion, at an early stage the projects were mostly “wide focused”, but improvements in A&F design and delivery have been introduced, and projects, still ongoing, will offer interesting insights on A&F effectiveness.
Keywords:
Subject: Public Health and Healthcare - Public Health and Health Services
1. Introduction
Among interventions aimed at changing health providers’ behaviors, “Audit & feedback” (A&F) has been emerging as one of the most promising as its effectiveness has been shown in different settings. In an A&F process, an individual’s professional practice or performance is measured and then compared to professional standards or targets [1,2].
Indeed, providing health professionals with structured reports on their performance can lead to quality improvement when process or outcomes indicators highlight potential problems concerning the delivery of specific health care interventions or the clinical outcomes achieved. However, many factors influence the yield of A&F interventions, including the context in which they were applied, the type of targeted behaviors, and how they are structured and delivered [1,3,4].
A&F includes a wide range of interventions which differ in terms of type of quality indicators used, structure and frequency of the report feedback to health professionals. While it is known that A&F interventions can improve clinical practice in general, it is much less clear how they should be applied in health care organizations. Recommendations on how an A&F system should be conceived and developed to be able to fully exploit its potential have been provided by experts in the field [4,5,6]. However, the actual applicability of these recommendations - based on the available evidence, mostly based on UK and North America experiences, as well as on assumptions from theories of behavior change and common sense - should not be taken for granted. A&F are indeed complex interventions strongly influenced by the context, whose implementation poses several challenges in terms of competences and skills to be involved in its conceptualization and design, and technical problems to be considered.
In Italy, the Ministry of Health launched in 2019 a national research program (NET-2016-02364191) [7,8], aimed at exploring the worth of A&F interventions, with participating Institutions in seven regions (Lazio, Friuli-Venezia Giulia, Piedmont, Emilia-Romagna, Lombardy, Calabria, Sicily) conducting projects applying A&F initiatives in different settings (Table 1).
Aim of this paper is to outline how A&F interventions under the EASY NET project were designed at an early stage, to explore the extent to which current recommendations on desirable characteristics of an "ideal" A&F procedure were adopted.
2. Materials and Method
In the framework of the EASY-NET national research program a template was developed with the aim of collecting information on how the different A&F interventions were designed in order to meet the specific needs of each project.
The template was divided into six sections addressing the following components of the A&F intervention to be developed:
- description of the working group composition (i.e., competencies, skills and clinical and organizational responsibilities represented in the team in charge of designing the A&F intervention);
- targeted clinical behavior(s) (i.e., the clinical behavior(s) to be changed through the A&F intervention);
- selected indicators and their informative sources;
- the feedback procedures to be adopted (i.e., timing and frequency of the reports, as well as their structure);
- actions (if any) expected from the targeted health professionals (i.e., what health professionals were supposed to do or act on after feedback delivery);
- other intervention(s) if any, to be carried out along with the A&F procedure to sustain / reinforce its impact.
Information gathered through the template was then classified into four main topics (nature of the desired action - i.e. the targeted behavior-, type of data available for feedback, feedback display and feedback delivery), in line with the categorization used by Brehaut et al. 2016 for their recommendations [4].
3. Results
3.1. Nature of the desired action
In three of the nine projects (WP3A, WP3B, WP6) the A&F procedure was aimed at changing a targeted and clearly identifiable clinical behavior. In six circumstances (WP1A, WP1B, WP2, WP4, WP5, WP7) A&F was generically aimed at stimulating the attention to the quality of care provided to specific categories of patients. Moreover, the type of actions that intervention research teams expected from the recipients as consequences of the A&F procedure seems to be generic, being reported as “dialogue among colleagues” or “promotion of comparison among peers”. All the projects identified clinicians (GPs and/or specialists) as the recipients of the information. Managers and other professionals with organizational responsibilities were explicitly considered in almost all projects (except for WP4, WP5, WP6).
3.2. Nature of the data available for feedback
Indicators were planned to be developed in all projects from administrative databases. Moreover, in some cases ad hoc data collection was planned, using dedicated web-based databases in four cancer audits, self-completed questionnaires in one study, and interviews in two studies. Overall, the mean number of process/outcomes indicators considered in the A&F projects was 27 range (9 – 61). Most of them (75%) were process indicators, such as measures of adherence to guideline recommendations on therapies and clinical examinations. Indicators designed to describe the volume of activity (i.e. the number of patients/interventions included in the pathway/study, hospital ward or ED patient stay and their characteristics) were included in all the studies. Outcome measures (i.e., hospitalizations, length of hospital stay, mortality and complications rates) were relatively less considered, accounting for 25% of the whole number of indicators used.
The reference used as comparators were derived from recipient performance (changes over time), formal guidance or a peer group (mean performance of similar persons or organizations). In all the projects reference standards were drawn from scientific literature (i.e. international standards), from locally available data (average regional data, comparison with professionals of the same Primary Care Department) or from national law.
In 4 projects (WP1A, B, WP4, WP7) the provision of feedback was scheduled half yearly, but other frequencies were reported: annually (WP2, WP3B), every 3 months (WP3B), on demand (WP3A). One project did not indicate the feedback frequency.
The reference time varied from 6 months (WP1A, WP6 projects), 12 months (WP1B, WP7 projects) in the collected projects. In the other projects the reference time was not explicitly reported.
3.3. Feedback display and delivery
All the feedback strategies, according to the different frequency and timelines chosen for each project, allowed access to aggregated data that can be displayed through graphs and tables. These data were planned to be compared with reference standards reported in scientific literature or between different areas. Different feedback sending options were chosen: web platforms with access credentials (WP1, WP3 A), e-mails (WP6, WP7), workshops (WP2) and individual meetings (WP3B). While in three initiatives the use of economic incentives to encourage clinicians’ participation was mentioned, no additional intervention along with the A&F procedure was planned in the other regional projects. Main intervention features have been reported in Table 2.
4. Discussion
In this paper, we describe a wide experimental introduction of A&F in healthcare organizations in Italy, promoted by a national program specifically aimed at spreading these quality improvement strategies and assessing their impact on quality of care. The information collected at the beginning of the project through the survey carried out on the nine regional WPs (projects) gave us the opportunity to explore how these types of intervention were designed when applied in the context of health care organizations.
Three main relevant issues emerged from the original design of the A&F interventions developed by the regional projects.
The first issue concerns the specific aim of the intervention itself. According to Brehaut, intervention’s research teams should recommend actions that were consistent with established goals and priorities. Therefore, these should be explicit, specific, time-bound, recipient-defined, challenging but also attainable, with room for improvement and over which the recipient has control. Moreover, specific rather than general actions should be recommended to be more effective. Of course, compliance with these recommendations implies clear identification of a clinical behavior to be targeted and hopefully changed through the A&F intervention.
In 7 out of 9 analyzed projects the objectives of the A&F intervention were quite broad, with multiple purposes, aimed at promoting greater attention to the improvement of the quality of care in the clinical area considered. It was assumed that A&F, providing a broad description of the quality of care through a set of indicators, could have a general positive impact offering opportunity for discussion to the multidisciplinary clinical communities involved in the provision of care. In such a context, A&F seems implicitly seen more as a “clue” keeping together through the information provided the different stakeholders, than a tool aimed at changing few, well specified behaviors.
The second issue is about process and outcome indicators. The high number of process and outcomes indicators used in the regional projects described seems to be consistent with this broad, unfocused approach. If the goal is to provide information on the whole patterns of care in a clinical area, rather than changing a targeted behavior, several indicators are required. Of course, this raises several issues concerning the availability of timely and valid data and the extent to which recipients of reports based upon such a high number of indicators could be overwhelmed, rather than guided, by the amount of information received.
As far as indicators are concerned, it is worth noting the almost exclusive reliance on administrative databases. It is well known that administrative database is indeed a valuable tool when it comes at assessing quality of care, however there are relevant dimensions of quality that cannot be fully explored using only that source of information. Assessment of the appropriateness of use of health care interventions requires detailed clinical information on individual patients’ characteristics typically missing in administrative databases. The exclusive reliance on administrative databases also limits the timeliness and frequency of the feedback, being the reporting system conditioned on the actual availability of the data.
Overall, such unbalance between administrative and clinical databases is at odd with the extensive penetration of ITC technologies in health care organizations. Electronic medical records and laboratory databases, which are available in most health care settings, seem to be tools designed and used mainly for the management of single patients or for administrative management. Their potentials are still far from being fully exploited for quality assessment and improvement purposes. Electronic medical records are indeed much richer than administrative databases of detailed clinical information. That information can be made available to recipients much more frequently and timely. The relative underrepresentation in our sample of ICT experts in the teams responsible for the design of the A&F interventions points out the extent to which this aspect seems to be still overlooked.
The third issue concerns the feedback report strategy. In an A&F intervention, the research teams should keep in mind that feedback is more effective if a summary message and a visual display are both included and linked conceptually and visually. Moreover, providing feedback in more than one way (for example, combining spoken words and picture) and minimizing extraneous cognitive load (without overly complex information) for feedback recipients can positively affect the efficacy of the intervention.
When deciding on how to deliver a feedback intervention, according to the 15 suggestions, the A&F research teams should consider different actions [4]. Among these, actions worth considering are: addressing barriers to using feedback to reach the intended target audience; providing short, operational messages with optional information available for interested recipients; addressing credibility of the information, such as with the help of supervisor or colleague; preventing defensive reactions to feedback; and building feedback through social interaction, rather than passively receiving it. Little correspondence of these suggestions was found, at least in the planning phase of the projects analyzed.
Our account of how health care organizations in Italy approach the challenge of designing and implementing audit and feedback interventions obviously has limitations. Firstly, we provide a description on how A&F interventions have been conceived and planned at a very early stage. It is reasonable to assume that some changes in their structure may have occurred during the actual implementation phase, as some preliminary findings seem to show [9,10,11,12]. In addition, these projects are still ongoing, and therefore we do not have yet information on their actual impact on clinical practice, but many improvements have been possible, including through collaboration with the A&F International Meta lab, established under EASYNET. Furthermore, while health care organizations from seven regions in Northern, Central and Southern Italy were involved in our survey and the results should be considered representative of the Italian context, the extent to which they can be actually considered generalizable to other countries is questionable.
5. Conclusions
From the description of the projects designed in the different regional settings involved in the Italian National A&F Program EASY-NET, it can be concluded that these interventions, at least at an early stage, are mostly intended as "generic reminders" to generally improve the quality of care, rather than interventions aimed at changing specific clinical behaviors as described by the prevailing research literature and expert recommendations. The changes introduced during the course of the projects (some forced due to the COVID pandemic) and the final results obtained will offer interesting insights into the ultimate effectiveness of the different A&F strategies.
Author Contributions
“Conceptualization, R.G., N.A. and C.A.; methodology, R.G, F.V. and C.A..; validation, G.C., A.G.d.B., M.D., and A.A.; formal analysis, F.V.; investigation, C.A.; resources, R.G.; data curation, C.A.; writing—original draft preparation, F.V., R.G.; writing—review and editing, C.A., N.A., GC and A.A; visualization, M.D., A.G.d.B.; supervision, R.G., N.A.; project administration, R.G.; funding acquisition, N.A. All authors have read and agreed to the published version of the manuscript.” Please turn to the for the term explanation. Authorship must be limited to those who have contributed substantially to the work reported.
Funding
Please add: “This research was funded by the EASY-NET program co-funded by the Ministry of Health and the participating Regions (NET-2016-02364191).
Informed Consent Statement
“Not applicable” here.
Data Availability Statement
“Not applicable” here
Acknowledgments
This work was produced as part of the activities of the EASY-NET research group "Audit & Feedback. Effectiveness of Audit & Feedback strategies to improve healthcare practice and equity in various clinical and organizational settings (EASY-NET)" (project code: NET-2016-02364191), funded by the Ministry of Health and co-funded by the participating regions (Lazio, Friuli Venezia Giulia, Piedmont, Emilia-Romagna, Lombardy, Calabria, Sicilia).
Conflicts of Interest
“The authors declare no conflict of interest.”
References
- Ivers, N. M.; Sales, A.; Colquhoun, H.; Michie, S.; Foy, R.; Francis J. J.; and Grimshaw J. M.. No more 'business as usual' with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci 2014, 9, 14. [CrossRef]
- Grimshaw, J. M.; Ivers, N.; Linklater, S.; Foy, R.; Francis, J. J.; Gude, W. T.; and Hysong, S. J.; Audit and Feedback MetaLab. Reinvigorating stagnant science: implementation laboratories and a meta-laboratory to efficiently advance the science of audit and feedback." BMJ Qual Saf 2019, 28(5), 416-423. [CrossRef]
- Ivers, N. M.; Jamtvedt, G.; Flottorp, S.; Young, J. M.; Odgaard-Jensen, J.; French, S. D.; et al. Audit and feedback: effects on professional practice and healthcare outcomes." Cochrane Database Syst Rev 2012, (6), CD000259. [CrossRef]
- Brehaut, J. C.; Colquhoun, H. L.; Eva, K. W.; Carroll, K.; Sales, A.; Michie, S.; et al. (2016). Practice Feedback Interventions: 15 Suggestions for Optimizing Effectiveness. Ann Intern Med 2016, 164(6), 435-441. [CrossRef]
- Brown, B.; Gude, W. T.; Blakeman, T.; van der Veer, S. N.; Ivers, N.; Francis, J. J.; Lorencatto, F.; et al. Clinical Performance Feedback Intervention Theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implement Sci 2019 14(1), 40. [CrossRef]
- Foy, R.; Skrypak, M.; Alderson, S.; Ivers, N. M.; McInerney, B.; Stoddart, J.; et al. Revitalising audit and feedback to improve patient care." BMJ 2020, 368, m213. [CrossRef]
- “EASY-NET network program.” 2019. [Online]. Available: https://easy-net.info 326.
- Acampora, A.; Deroma, L.; Ciccone G, Marchesini Reggiani G, Marenzi G, Venturella R, Bramanti P, Grilli R, Di Martino M, Spadea T, De Fiore L, Agabiti N; Gruppo di lavoro EASY-NET. Il programma di rete EASY-NET: razionale, struttura e metodologie [The EASY-NET research programme: background, structure, and methodology]. Epidemiol Prev. 2023 47(1-2), 80-89.
- Angioletti, C.; Pinnarelli, L.; Colais, P.; Angelici, L.; de Mattia, E.; Davoli, M.; de Belvis, A.G.; Agabiti, N.; Acampora, A.; Audit and Feedback in the Hospitals of the Emergency Networks in the Lazio Region, Italy: A Cross-Sectional Evaluation of the State of Implementation. Healthcare (Basel). 2022, 11(1). [CrossRef]
- Angelici, L.; Angioletti, C.; Pinnarelli, L.; Colais, P.; de Mattia, E.; Agabiti, N.; Davoli, M.; Acampora, A.; EASY-NET Program: Methods and Preliminary Results of an Audit and Feedback Intervention in the Emergency Care for Acute Myocardial Infarction in the Lazio Region, Italy. Healthcare (Basel) 2023 11(11). [CrossRef]
- Piovano, E; Pagano, E; Del Piano, E; Rinaldi, F; Palazzo, V; Coata, P; Bongiovanni, D; Rolfo, M; Ceretto Giannone,L; Veliaj, D; Camanni, M; Puppo, A; Ciccone, G; ERAS- Gyneco Piemonte group. Implementation of the ERAS (Enhanced Recovery After Surgery) protocol for hysterectomy in the Piedmont Region with an audit&feedback approach: Study protocol for a stepped wedge cluster randomized controlled trial. A study of the EASY-NET project. PLoS One. 2022;17(5). [CrossRef]
- Pagano, E; Pellegrino, L; Rinaldi, F; Palazzo, V; Donati, D; Meineri, M; Palmisano, S; Rolfo, M; Bachini, I; Bertetto, O; Borghi, F; Ciccone, G; ERAS Colon-Rectum Piemonte study group members. Implementation of the ERAS (Enhanced Recovery After Surgery) protocol for colorectal cancer surgery in the Piemonte Region with an Audit and Feedback approach: study protocol for a stepped wedge cluster randomised trial: a study of the EASY-NET project. BMJ Open. 2021 ;11(6). [CrossRef]
Table 1.
List of EASYNET regional projects (Work Packages, WPs).
| Region | Project title |
|---|---|
| WP1 A Lazio | Comparative evaluation of the effectiveness of Audit and Feedback (A&F) strategies to improve integrated care pathways for chronic conditions |
| WP1 B Lazio | Comparative evaluation of the effectiveness of Audit and Feedback (A&F) strategies to improve integrated care pathways for acute conditions |
| WP2 Friuli-Venezia Giulia | Prospective Audit and Feedback Approach: effectiveness in improving clinical care and in reducing avoidable health differences in emergency |
| WP3 A Piedmont | Clustered randomized controlled study - stepped wedge - on the implementation of the Enhanced Recovery After Surgery (ERAS) protocol supported by an A&F strategy in general and gynecologic surgery |
| WP3 B Piedmont | Prospective Regional Audit and Feedback on Ovarian and Bladder Cancer Treatment in Piedmont |
| WP4 Emilia-Romagna | Effectiveness of Audit and Feedback interventions for the improvement of health care in Type 2 Diabetes mellitus and Chronic Heart Failure |
| WP5 Lombardy | Effectiveness of Audit and Feedback strategies to improve health practice and equity in patients with heart disease |
| WP6 Calabria | Evaluation of the effectiveness of a prospective Audit and Feedback approach to improve health practice and reduce the rate of caesarean sections |
| WP7 Sicily | Effectiveness of a new clinical audit and clinical model as part of a pathway of high reliability in health care |
Table 2.
Interventions features.
| Project | Aim – Changing Behaviour | Recipients | Indicators (Source And Type) | Comparators | Timing Of Feedback | Display-Delivery Of Feedback |
|---|---|---|---|---|---|---|
| WP1 a) Lazio Chronicity | Generic, focused on patient empowerment and continuity of care improvement | Clinicians (General Practitioners) Local Unit Managers |
Administrative Clinical database Process and outcome |
Explicitly identified (Regional mean, district mean) |
6 months | Oral (meetings) Written (report) |
|
WP1 b) Lazio Emergency |
Generic, focused at improving care practice through knowledge of the quality standards to be guaranteed | Clinicians (cardiologists, neurologists, specialists in emergency area) Managers /Other professionals with organizational responsibilities |
Administrative Clinical database Process and outcome |
Reference standard (regional mean) | 6 months | Oral (meetings) Written (report) |
| WP2 FVG | Generic, focused on clinical, features, structural factors, and policies improvement | Clinicians (cardiologists, neurologists, neurologists, specialists in emergency care) Nurses Managers / Other professionals with organizational responsibilities |
Administrative data Process and outcome |
Reference standard | 12 months | Oral (workshop) Written (report) Individual (virtual reality) |
| WP3 a) Piemonte colorectal resections and Hysterectomy | Targeted, focused on ERAS protocol application | Clinicians (oncologists -multidisciplinary teams) Managers / Other professionals with organizational responsibilities |
Administrative Clinical databases Process and oucome |
Reference standard (regional mean) | 3 months / 2 months / 6 months depending on the type of cancer pathway of care |
Oral (meetings) Web – site Written (report) |
|
WP3 b) Piemonte ovarian and bladder cancer |
Targeted, focused on Oncological Network recommendations and international guidelines | Clinicians (oncologists -multidisciplinary team) Managers / Other professionals with organizational responsibilities |
Administrative Clinical databases Process and Outcomes |
Reference standard (regional mean) | 12 months | Oral (meetings) Web – site Written (report) |
| WP4 EMILIA ROMAGNA | Generic, focused on care integration and coordination improvement | Clinicians (diabetologists, cardiologists -multidisciplinary teams) | Administrative Clinical databases Process and Outcomes |
Reference standard (regional mean) | 6 months | Written (report) |
| WP5 Lombardia | Generic, focused | Clinicians (cardiologists) | Administrative Clinical databases Process and Outcomes |
Reference standard (regional mean) | not available | Written (report) |
| WP6 Calabria | Targeted, focused on improving the appropriateness of indications for caesarean section | Clinicians (gynecologists) | Administrative Process and Outcomes |
Reference standard (regional mean) |
Two weeks | Oral (meetings) Written (report) |
| WP7 Sicilia | Generic, focused on improving process and outcomes measures |
Clinicians (cardiologists, neurologists, multidisciplinary teams) Managers / Other professionals with organizational responsibilities |
Administrative Process and Outcomes |
Reference standard (regional mean) | 6 months | Oral (meetings) Written (report) |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
MDPI Initiatives
Important Links
© 2024 MDPI (Basel, Switzerland) unless otherwise stated