The Clinical Strategies Implementation Scale Revised (CSI-R). Fidelity Assessment of Resource Group Assertive Community Treatment ()
1. Introduction
Assertive Community Treatment (ACT) models are community based treatment and rehabilitation programs intended primarily for individuals with long-term mental illness [1]. The original variant was carried out by a purely psychiatric multidisciplinary health-care team where all members of the team were capable of working with all of the clients and where all team members could have an intensive clinical case management function [2]. The treatment is carried out in the client’s own immediate surroundings rather than at a clinic or other healthcare setting.
There are at present a number of different models for ACT and new experience and research results are constantly reported [3]. The approach introduced by Falloon [4,5] expanded the original family unit into a “resource group” including persons from the social network of the client. In this Resource Group model it is the client who determines what the overall objectives should be, but the resource group takes joint decisions about how to achieve them. The model has been given a number of different names but, since the model continues to be developed with an ever greater emphasis on the central position of the client through the participation of the Resource Group, the decision was to label the model the Resource group ACT or RACT [6].
The RACT model was described in detail [6] in a phenomenological study showing among other results that the RACT manual, which contains training programs, training protocols, organizational ideas, methodology for follow-up of results, and quality control, is of utmost importance for giving the case manager, jointly with the client, tools for the construction of an effective Resource Group. The fact that the client him/herself defines his/her own treatment goals, nominates those to be included in the Resource Group, and is trained by the case manager to be, if possible, the leader of the Resource Group, all seem to be crucial factors in the empowerment of the client. The empowerment of the client, in turn, appears to be the major driving force for successful treatment. The results of the study [6] also indicated that Resource Groups may be useful both to psychotic clients and clients with other types of diagnoses. Those findings are in line with a recent meta-analysis [7] which indicated a large effect-size for the “grand total measure” (Cohen’s d = 0.80). The conclusions of the meta-analysis were that the treatment of clients with Resource Group Assertive Community Treatment yields positive effects in regard to symptoms, functioning, and well-being for clients with psychoses and that the method may be useful for clients within the entire psychiatric spectrum.
In order to secure proper implementation fidelity, assessments are needed [8,9]. The DACT scale is the standard for fidelity assessment to the original ACT principles on service level [10]. Over time, adjustments to the scale have been suggested, for instance the TMACT scale [11]. These tools can also be used to assess adherence by teams to ACT principles in RACT praxis. However, these instruments do not assess the adherence to specific strategies and are not intended for fidelity assessments in individual cases.
The Clinical Strategies Implementation scale (CSI) was designed to be used by external reviewers in order to help providers measure the extent to which evidencebased strategies had been implemented in the treatment of persons with schizophrenia spectrum disorders according the model advocated by Falloon [12]. Due to the further development of the model, with the strong emphasis on the Resource Group as a major strategy, a revision of the CSI was necessary. Also, the CSI is a resource-demanding fidelity assessment as it requires external expert reviewers. It was therefore important that the revision also included the need to make the revised scale suitable as an instrument to be used by students and practitioners of the RACT model themselves as a quality tool in self-monitoring of RACT fidelity.
Aims
The present investigation had two aims. The first was to conduct a revision of CSI and to examine the revised instrument (CSI-R) in terms of interrater reliability (Study I). The second aim was to compare assessments of CSI-R made by experienced assessors and made by students in case management (Study II) in order to determine whether the instrument has validity even when more inexperienced persons are using it.
2. Study I: Methods (Revision of CSI)
2.1. Participants
Participating CSI-R raters were nurses, psychiatrists and psychologists, with over five years’ experience of practising RACT. They were recruited from four different psychiatric organisations in Sweden. Only raters who took part rating of at least 12 of the 15 cases were included in the study (N = 6). Cases to be rated (N = 15) with the CSI-R were presented by their case-managers from three outpatient Community Mental Health teams in the Göteborg area, ranging from being well-functioning in active rehabilitation to highly impaired persons with intensive care.
2.2. Instrument
CSI-R (Clinical Strategies Implementation scale—Revised). The CSI-R is a development of the original CSI instrument [12]. The CSI consists of nine ordinal scales, devised to measure key aspects of treatment strategies that have been associated with clinical and social recovery: goaland problem-oriented assessment, medication strategies, assertive case management, mental health education, caregiver-based problem solving, living skills training, psychological strategies for residual problems, crisis prevention and intervention, and booster sessions. There are defined operational criteria for each rating level from −1 (wrong-doing) to +4 (fully achieved). Interrater reliability for experienced trained raters was 0.93 to 0.99. Validity was established by significant correlations with measures for impairment, disability, functioning, work activity, and an index of recovery. Each treatment strategy item was given a consensus weight though an international focus group of 12 experienced clinicians as shown in Table 1. Translation to Swedish was done in dialogue with clinicians, familiar with its contents, having used the original CSI in English. Formal retranslation was therefore not necessary.
The adjustments of the CSI in order to better cover the RACT praxis in Sweden, as pointed out in the Introduction, i.e. the creation of the CSI-R, included the following areas (compare Table 1).
Due to the Swedish health service model context, the “Physical health management” item was split from the “Assertive case management” item, as somatic care is routinely carried out by general practitioners (or specialists when needed). Also, the “Booster session” item was not considered appropriate in the Swedish service model, as its inherent content was felt to be part of the proper handling of the respective strategies. Due to the emerging evidence of the importance of the Resource Group in enhancing the effects of RACT [6,7] the original item called “Extended Family & Social Network Intervention”, was changed to “Resource Group Management”.
Item construction has also been revised. As the CSIoriginal criteria ladders were sometimes found difficult for clinicians to interpret, the CSI-R item criteria were compressed into text vignettes above each item as a guide to what the requirements are to be fulfilled for the highest rating (+4) are. The negative rating −1 (counterproductive activities) was incorporated into the 0 rating, as it was not wanted that “non-doing” (rating 0) should to be considered better than “wrong-doing” (−1). Finally, the scales of all items were given points 0 - 4. Rating 4 is given when all elements in the vignette are full-filled, rating 0 when none are achieved, the intermediate scale steps ratings are used, in principle, with almost all elements fulfilled (rating 3), some elements (2), only initial fulfillment of one or few elements (rating 1). Weights scores for the items of the CSI-R were kept from the CSIexcept the weight for “Living skills training” and for new or revised items that got their weights through clinical expertise consensus discussions.
The construction of the CSI-R was done through iterative discussions between authors and several groups of experienced clinicians until consensuses were reached and face validity was deemed acceptable. The resulting new CSI-R in overview is found on the right in Table 1.
Finally, the scale introductory text and item text was phrased for use in two ways, one for the use of external reviewers and one for clinicians’ own assessment of RACT fidelity. The CSI-R was field tested for validity in a series of yearly peer reviews at services practicing RACT based case management, and the wording was successively improved in an iterative process, until face validity was deemed satisfactory.
2.3. Procedure
Raters who participated in the study to examine the revised CSI-R instrument were firsts trained in using it. To ensure that ratings would be carried out with good reliability, deviating ratings were discussed in depth to clarify if rating differences were caused in the understanding of the scale steps, or merely differences in the interpretation of the case presentations. The discussion went on until raters were in agreement of what was the appropriate rating.
Following the training, the raters have met on several occasions when about four cases a time were presented by the cases’ case-managers. Raters could ask for further information from the case manager needed to do a CSI-R
Table 1. Overview of the original CSI and the adjusted CSI-R covering core elements of RACT.
rating. The six raters included in the study where those who had had the opportunity to attend and rate between 12 - 15 cases of severe mental illness. Ratings score sheets were handed to the second author for analysis. Data obtained were analyzed for scale internal consistency and interrater reliability.
3. Study I: Results (Revision of CSI)
The data obtained from six raters who had participated in rating at least 12 cases showed that internal consistency of the CSI-R was strong, with Cronbach’s Alpha 0.89. Correlation between individual raters’ total score and the pooled mean of the six raters was strong (r between 0.80 - 0.98). Conclusive results on item level can not be presented due to the limited number of cases. Thus, in the following study, comparisons between students’ ratings and expert ratings on total sum level should be safe, whereas comparisons on item level are not used in this study.
4. Study II: Methods (CSI-R Expert Ratings and Students Ratings)
4.1. Participants
Students who participated with CSI-R ratings were 91 newly trained RACT praxis trainees during their final, 3rd term. They were primarily nurses and social workers employed in social welfare care or psychiatry. Each of them followed one case for eighteen months, i.e., the client which they had been assigned during training (n = 91). Five of the six external expert raters from Study 1 participated in the study.
4.2. Instrument
CSI-R as described in Instruments section Study I.
4.3. Procedure
In order to monitor progress in attaining knowledge and skills an external audit each term was part of the education program. In the weeks before an external audit, students had made their CSI-R ratings. Rating protocols were without any information that could identify a patient, and were kept in sealed envelopes. Thus, the external auditors were blind to the students’ ratings, when they made their CSI-R rating based on a 30 minute long interview with the trainee. Four of the expert raters independently assessed four different groups of students, while one of the external raters had to assess two student groups. There was no scrutinizing of case notes by the reviewers. Data were subsequently analyzed by the authors regarding rating agreement between case managers and experts. For overview of participants in Study II, see Table 2.
Table 2. Education groups, trainee participants (n) and fidelity group scores in percentage (%).
5. Study II: Results (CSI-R Expert Ratings and Students Ratings)
The total raw sum Spearman correlation between experts and trainees was at 18 months rho = 0.68 (p < 0.001). Regarding ratings of the nine individual items, frequency tables for both case managers and expert raters showed very small distributions, and cross tabulations revealed that identical ratings were given in about 60% of the ratings, and in about 30% of the cases ratings were near identical, with a difference in rating of plus or minus one scale step. These findings are summarized in Table 3. Only items 4 (Carer-based stress management) and 7 (Specific psychological treatments) had notable disagreements in some 10% - 20% of the cases.
The level of rating agreement between case managers and experts was tested with Kendall’s Tau and Cohen’s Kappa (see Table 4). A general tendency of fair agreement was found, with values between 0.3 and 0.5, though figures for a few items showed some weaknesses, which is understandable because of the limited material, many tied ranks and small distribution ranges.
Finally, the weighted scores of the total CSI-R programme fidelity percentage in the six education groups were calculated, and are shown on the right in Table 2.
6. Discussion
The Resource group ACT (RACT) model, outlined in the Introduction, contains mechanisms for further strengthening of patients’ social and professional network participation. Emerging evidence indicate that the RACT model yields positive effects for clients as compared with traditional praxis. This development called for a development of an instrument for assessment of program fidelity to the RACT model.
The present investigation had two aims. The first was to conduct a revision of a previous CSI scale to better suit RACT, and to examine the revised instrument (CSIR) in terms of interrater reliability (Study I). The second aim was to compare assessments of CSI-R made by experienced assessors and by case management students
Table 3. CSI-R* rating agreements between trainees and external auditors at 18 months.
*Item 1-9 = Item contents, see Table 1.
Table 4. CSI-R rating correlations (Kendall’s Tau) and rating agreements (Cohen’s Kappa) between trainees and external auditors at 18 months.
in order to determine whether the instrument has validity even when more inexperienced persons are using it (Study II).
The revision reported in Study I included minor changes of three of the 9 items, and particularly, the development of a specific item for Resource Group strategy. The new CSI-R scale had good internal consistency, with Cronbach’s Alpha 0.89. Correlation between individual expert raters’ total score and the pooled mean of the raters was strong (r between 0.80 - 0.98).
Conclusive results on item level can not be presented due to the limited number of cases. Rater agreement may thus be questionable for some items. It calls for further studies to clarify if construct and/or face validity may still need improvement.
In Study II, comparing CSI-R ratings by students in final stages of RACT praxis education and ratings by experts, the total raw sum Spearman correlation was acceptable with rho = 0.68 (p < 0.001). Regarding ratings of the nine individual items, further studies are needed to clarify this in detail, as the present findings are based on a limited material, with many tied ranks and small distribution ranges.
It also needs to be emphasized that the studies have been carried out with serious mentally ill cases representing not only schizophrenia spectrum, but also for instance personality disorders, and dual diagnosis cases. The CSI-R appears equally informative regardless of diagnosis. This is important as a previous study [13] as well as praxis [6] supports that RACT model is advantageous also for other serious mentally ill persons. A recent RACT meta-analysis [7] also indicated that this is not limited to the Swedish context, and it is therefore suggested that its fidelity instrument, the CSI-R can be considered elsewhere.
In summary, the new CSI-R scale was shown to have acceptable internal consistency and interrater reliability. Also, it was shown that comparisons between students’ total sum ratings were reasonably comparable to experts’ ratings. Thus, a first version of an instrument for fidelity assessment of RACT is available, which may be used both for external review and for continuous self-monitoring of praxis fidelity by inexperienced raters.
NOTES