ACAT

You are here: Home > Reformulation > Summer 2011 [Issue 36] > An audit of Goodbye Letters written by clients in Cognitive Analytic Therapy

An audit of Goodbye Letters written by clients in Cognitive Analytic Therapy

McCombie, C., Petit, A., 2011. An audit of Goodbye Letters written by clients in Cognitive Analytic Therapy. Reformulation, Summer, pp.42-45.

Abstract

In this paper we describe an audit of client-written Goodbye Letters in Cognitive Analytic Therapy (CAT), with the aim of using the letters as a tool for evaluating therapy efficacy and adherence to the core Vision and Values of Surrey and Borders Partnership NHS Foundation Trust. Five CAT therapists designed an audit tool specifically for this purpose, and thirty client-written Goodbye Letters were scored by two independent raters. Most letters scored highly, suggesting that therapy was generally effective and that the Vision and Values of the Trust are met within the therapy. However, inter-rater reliability was weak, leading to the conclusion that while the audit tool may be valuable, there are limitations in its use in the current form, as interpreting the very personal letters in terms of a 19 point scale is challenging.

Introduction

An almost unique1 feature of CAT is the letters written by the client and therapist as part of therapy. The therapist-written Goodbye Letter is intended as a summary of “what has been achieved and of what remains to be done” (Ryle, 1997, p. 122), and offering reflection on the therapeutic relationship. The client’s Goodbye Letter reflects on how they feel about the progress of therapy, what they have learned, and areas where they feel there is further work needed. The client-written Goodbye Letter also allows space to reflect on and acknowledge the ending of therapy and the loss of the therapeutic relationship. Hamill, Reid, and Reynolds (2008) found that clients value writing a Goodbye Letter, as it helps them review therapy and realise and restructure their thoughts.

As part of the new Mental Health Strategy, the practice of using outcome measures is moving more towards collecting meaningful information rather than just meeting targets. This means that client feedback is becoming more valuable in service improvement, and there is greater demand for outcome measures that look at more than just symptom improvement. Client-written Goodbye Letters encompass a variety of subjects, including client satisfaction and therapy efficacy, and each letter is personal to the writer, so there is good potential for client-written Goodbye Letters to become a useful tool in service evaluation.

CAT naturally encompasses the vision and values of the Trust [Surrey and Borders Partnership NHS Foundation Trust] – the values of Involve Not Ignore; Create Respectful Places; Open, Inclusive, and Accountable; and Treat People Well. Figure 1. shows these core values and their components. The aspects of ‘Involve Not Ignore’ and ‘Treat People Well’ are engrained in the concept of CAT, as CAT is based on collaborative work between the therapist and client, particularly when producing the Reformulation and amending it. The values of ‘Create respectful places’ and ‘Open, inclusive, and accountable’ are also included in the CAT process, as CAT allows the client to talk openly about their problems in a respectful and trustworthy space, and relationship between client and therapist in CAT is an important part of therapy, particularly in uncovering the reciprocal roles enacted by the client. Goodbye Letters written by the client, with their reflection on therapy from the client’s point of view, could therefore be useful in assessing whether teams are meeting the Vision and Values of the Trust.
Figure 1. Surrey and Borders Partnership NHS Foundation Trust Vision and Values.

The audit was therefore conducted with two main aims – to assess the efficacy of CAT from the client’s point of view, and to assess whether the core vision and value goals of the Trust are being met within CAT therapies in the Trust. The audit is intended to investigate whether the client-written Goodbye Letter can be used as an effective form of outcome measure or source of client feedback, and whether clients feel this type of therapy has been effective and valuable to them.

Method

Five Cognitive Analytic Therapists devised a tool for rating Goodbye Letters written by clients. They took the Trust’s Vision and Values document and summarised the key factors, and discussed important concepts of CAT. The tool has 19 questions, 8 relating to the Vision and Values of the Trust, and 11 relating to core concepts of CAT. The questions were designed to find out whether certain Trust values and concepts of CAT were mentioned in the letter. The questions could be scored 1 if it was mentioned or implied that the question had a positive answer, zero if the subject of the question was not mentioned, and minus one if it was mentioned or implied that the question had a negative answer. Sample items are ‘Felt listened to or heard and not ignored’, ‘Felt safe to be themselves in therapy’, ‘Indicate helpful adoption of alternatives to recognised difficult patterns of interaction’, and ‘Now has a more realistic take on what brought them to therapy’.

30 client-written Goodbye Letters were collected from four CAT therapists. Each therapist was from a different team: a Psychotherapy Department, a Primary Care Mental Health Team, a Community Forensic Service, and a Learning Disabilities Team. The letters were then given to raters who were not familiar with the therapy or the clients, to avoid biases in the rating. Two raters went though each letter independently, reading the letters carefully and scoring them according to the audit tool.

Results

The results show that scores are relatively high across all teams involved in the audit. Table 1 shows the mean scores across all teams and sections in the audit tool. Figure 2 shows the correlation between scores by both raters.

Figure 2 shows a scattergram of the scores produced by each rater. This shows that there is a weak-medium strength positive correlation between the two raters’ scores. A bivariate correlation was conducted on the two sets of scores (one from each rater) to establish whether there was sufficient inter-rater reliability. The correlation between individual scores was r = .152, p < .001. The correlation between total scores was r = .28, p > .05. Removing letters from the analysis with large differences (6 points or more) between the two scores (of which there were 5), the correlation between the two sets of scores was r = .4, p = .027. This is a significant correlation with a weak-medium strength.

A bivariate correlation was also conducted on the scores from the Trust Vision and Values and CAT features parts of the scale separately. For trust values, the correlation between total scores was r = .290, p > .05, and the correlation between individual scores was r = .198, p = .001. For CAT features, the correlation between total scores was r = .283, p > .05, and the correlation between individual scores was r = .139, p < .05. Neither of these are strong correlations.

Discussion

Scores on the audit tool were mostly high, indicating that therapy was generally effective at achieving its goals. Clients were largely able to reflect on their experiences, what had brought them to therapy and what they were now doing differently to try and reduce their problems. Most clients were satisfied with the therapy, came away from it with better understanding of their problems and how to deal with them, were able to internalise the therapeutic relationship and acknowledged the loss of the therapeutic relationship. The audit has also shown that Trust Vision and Values criteria are generally being met within CAT, with most clients mentioning several key values, including writing that they felt safe and listened to.

The audit tool was straightforward to use, and covered a representative selection of both the Trust’s values and the core features of CAT. The tool is therefore potentially valuable as an outcome measure in CAT, which meets the criteria of the new Mental Health Strategy of finding more meaningful outcomes. However, there was very poor correlation between the two raters’ scoring. While both raters agreed that they thought it was easier to score for the Trust values rather than the CAT features, analysis of the inter-rater reliability for the two parts of the tool separately revealed poor correlation between scores in both parts.

Among the individual team scores, two teams stand out – teams three and four. Team 3 has a lower mean score than the other teams. This may be because it is a learning disabilities (LD) team, so the letters were very short and covered little. For this reason results are shown for the mean scores excluding team 3, as this would make the mean score more representative and valid.

Two of the seven LD letters were written by carers, this may mean that the letters do not necessarily accurately represent the views of the client. Whilst carers’ views are important, the measure was designed to measure clients’ views, and comparing both views in the same study would be difficult and is not the aim of this study. The inclusion of carers views may have resulted in team 3 having a higher mean score than is representative of client-written Goodbye Letters within the team.

Team 4 also stood out, due to scoring highly on both CAT concepts and Trust Vision and Values. This may be because clearer guidelines were given on writing the letter, resulting in thorough letters that cover most aspects of therapy, resulting in higher scores on the audit tool. It should be noted, however, that a high score does not necessarily indicate more effective therapy – a low score may merely indicate that a person did not mention certain points of the therapy in their letter.

There are several limitations to this audit. Firstly, the cases being treated in CAT are mostly highly complex. For this reason, it may be hard to meaningfully evaluate therapy efficacy based on 11 questions.

Secondly, the letters themselves are very personal and individual to both the client and therapist. There was a wide range in both depth and breadth of the letters, and clearly writing such a letter was easier for some than others. To utilise this tool effectively, clearer guidelines could be given to clients on how to write the letter. Hamill et al. (2008) found that there was anxiety associated with having to write a Goodbye Letter to the therapist, with clients unsure what to write, what was expected of the, and some amount of worry as to whether their letter would not be ‘good enough’. So clearer guidelines may relieve some of this anxiety and provide more useful service evaluation data, but this should not be done at the cost of the personalisation of the letters, as this is very much a strength of the letters.

Thirdly, the scale asks whether the letter ‘mentions or implies’ things. The raters felt that this was a weak point of the scale, as whether something is implied is extremely open to interpretation. Clients’ personalities come across through the letters, so it is perhaps understandable that people will respond to that and interpret things that differently. However, the lack of clarity over when to interpret things as implied is reflected in the poor inter-rater correlation. Providing standardised training in how to use the audit tool may increase inter-rater reliability.
Finally, the scoring system itself may be predisposed to yield little meaningful data. For example, a score of zero may mean that the client did not mention that particular item, rather than indicating that this feature was not present in therapy. Similarly, a score of minus one may indicate the client’s accurate reflection on the difficulties of therapy and acknowledgement of where work still needs to be done, rather than indicating that this was a bad or lacking aspect of therapy.

For example, several letters were scored minus one for question 14, ‘they are more hopeful about the future?”. The cases treated with CAT are often highly complex, so a change in hopefulness as well as changes in behaviour patterns may not be achieved in the short duration of the therapy, especially when considered alongside the long-term nature of many disorders. One aim of CAT is that clients leave therapy equipped with an understanding of their problems, and the tools and skills needed to make continued progress after the end of therapy, so hopefulness and other indicators of therapy efficacy may come later along the path of recovery than therapy.

So, for the reasons outlined above, the data yielded could be interpreted as being of limited value. However, as a basic evaluation of therapy efficacy, the scale may be useful, as high scores were still achieved by most letters. The scale could be used to flag up where there are low scores, for further investigation of the individual letters to see if there are any real issues with the therapy.

Conclusion

The audit has been successful in showing that therapy has been effective and that Trust Vision and Values have been met during therapy, despite some limitations with the audit tool. Scores are high for most teams, and where scores are lower there is a clear explanation as to why. This suggests that there is potentially much value in increasing the availability of this therapy type and in using a key part of the therapy, client Goodbye Letters, as a form of client feedback to inform service and individual therapist improvements. However, evaluating letters from LD clients is made difficult by the brevity of the letters and the limited content, and there are key issues with the tool that need to be addressed if it is to be more widely used.

Further research into improving the scoring system of the audit tool to make the resulting data more meaningful would be valuable. Research into the use of therapist-written Goodbye Letters as an outcome measure may be useful, as it would provide an outcome measure based on another perspective on the efficacy of therapy, and further information for service development.

Acknowledgements

This paper was written by two assistant psychologists from Surrey and Borders Partnership NHS Foundation Trust who volunteered to audit the Goodbye Letters. Catherine McCombie works in an adult mental health team and her supervisor Darren George co-wrote the paper. Aoife Pettit works in a learning disability team in Aldershot, and her supervisor Julie Lloyd also co-wrote the paper.

References

Hamill, M., Reid, M., & Reynolds, S. (2008). “Letters in cognitive analytic therapy: The patient’s experience.” Psychotherapy Research, 18, 573-583.

Ryle, A. (1997). Cognitive analytic therapy and borderline personality disorder: The model and the method. New York, NY: Wiley & Sons Ltd.

HM Government, Department of Health. (2011). No health without mental health: A cross-governmental mental health outcomes strategy for people of all ages. Retrieved March 8, 2010, from Department of Health website: http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_123766

Surrey and Borders Partnership NHS Foundation Trust. (2008). Vision and values. Retrieved March 8, 2011, from Surrey and Borders Partnership NHS Foundation Trust website: http://www.sabp.nhs.uk/aboutus/visionvalues

Appendix

Copy of the audit tool.

Audit of Goodbye Letters written by people to their therapist at the end of Cognitive Analytic Therapy

This audit is an analysis of the values expressed in the end of therapy letters written to their therapists by people who have completed a course in Cognitive Analytic therapy. The first 8 items reflect the Trust’s Vision and Values approach and the additional 11 items highlight some central features of C.A.T.

Scoring: 1 = Mentioned or implied 0 = Not mentioned -1 = Implied or mentioned that this was not so

In the letter, does the writer describe or imply that they:

Full Reference

McCombie, C., Petit, A., 2011. An audit of Goodbye Letters written by clients in Cognitive Analytic Therapy. Reformulation, Summer, pp.42-45.

Search the Bibliography

Type in your search terms. If you want to search for results that match ALL of your keywords you can list them with commas between them; e.g., "borderline,adolescent", which will bring back results that have BOTH keywords mentioned in the title or author data.

Evaluation of CAT in GP Practice
Baker, J., 2003. Evaluation of CAT in GP Practice. Reformulation, Spring, pp.16-17.

An Update on Microsupervision
Bennett, D., 1999. An Update on Microsupervision. Reformulation, ACAT News Summer, p.1.

The PSQ Italian Standardisation
Fiorani, C. and Poggioli, M., 2011. The PSQ Italian Standardisation. Reformulation, Winter, pp.49-52.

What are the important ingredients of a CAT goodbye letter?
Turpin, C., Adu-White, D., Barnes, P., Chalmers-Woods, R., Delisser, C., Dudley, J. and Mesbahi, M., 2011. What are the important ingredients of a CAT goodbye letter?. Reformulation, Winter, pp.30-31.

The CAT Articles Review
Knight, A., 2008. The CAT Articles Review. Reformulation, Winter, p.30.

Help

This site has recently been updated to be Mobile Friendly. We are working through the pages to check everything is working properly. If you spot a problem please email support@acat.me.uk and we'll look into it. Thank you.

Search