Evaluating Competene in Psychotherapy
Evaluating Competene in Psychotherapy
Evaluating Competene in Psychotherapy
136
medical knowledge, practice-based learning and improvement, interpersonal and communication skills,
professionalism, and systems-based practice (1). The
Residency Review Committee (RRC) for Psychiatry
has chosen an additional ve areas of psychotherapy
in which trainees in psychiatry are to be certied by
their training programs as competent: brief therapy,
cognitive-behavioral therapy, combined psychotherapy and psychopharmacology, psychodynamic
therapy, and supportive therapy (2). Some of these
therapies are closely allied to or derivative of others,
and there is considerable overlap in practice. This paper will explore issues affecting the dependable assessment of competence in several of these complex
and varied types of psychotherapy.
We know something about which aspects of these
ve psychotherapies are important for effectiveness
Academic Psychiatry, 27:3, Fall 2003
MANRING ET AL.
EVALUATING COMPETENCE
TABLE I.
A. Boundaries: The ability to 1) establish and maintain a treatment frame (e.g., time, space, outside agencies, outside
relationships, setting schedules and sticking to times), 2) establish and maintain a professional relationship, 3) understand and
protect the patient from unnecessary intrusions into privacy and condentiality, and 4) handle nancial arrangements with
patients in a manner appropriate to the treatment context.
B. Therapeutic Alliance: The ability to 1) establish rapport with a patient, 2) understand and develop a therapeutic alliance with
the patient, 3) recognize a variety of forms of therapeutic alliances, 4) enable the patient to actively participate in the treatment,
5) recognize and attempt to repair disturbances in the alliance, 6) establish a treatment focus, and 7) provide a holding
environment.
C. Listening: The ability to 1) listen nonjudgmentally and with openness, and 2) facilitate the patient talking openly and freely.
D. Emotions: The ability to 1) recognize and specically describe affects, 2) tolerate direct expressions of hostility, affection,
sexuality and other powerful emotions, 3) recognize and describe (to ones supervisor) ones own affective response to the
patient, and 4) recognize and tolerate ones uncertainties as a trainee in psychotherapy.
E. Understanding: The ability to 1) empathize with the patients feeling states, and 2) convey empathic understanding.
Use of Supervision: The ability to 1) establish an educational alliance with the supervisor, and 2) incorporate material discussed in
supervision into the psychotherapy.
F. Resistances/Defenses: The ability to 1) identify problems in collaborating with the treatment or the therapist, 2) recognize
defenses in clinical phenomena, and 3) recognize obstacles to change and an understanding of possible ways to address them.
G. Techniques of Intervention: The ability to 1) maintain focus in treatment when appropriate, 2) confront a patients statement,
affect or behavior, 3) assess readiness for and manage termination from treatment, 4) assess patients readiness for certain
interventions, and 5) assess the patients response to interventions.
138
competence on a continuum of increasing skill as described in the Dreyfus model of skill acquisition (17).
Table V illustrates these concepts. Most signicant is
the placement of competent as a skill level midway
between novice and expert. A trainee need not fully
master all the individual skills of psychotherapy to
be considered competent.
Once we agree on the skills necessary for competence in psychotherapy, how do we assess whether
trainees actually possess these skills? ACGME con-
TABLE III
MANRING ET AL.
structed a tool chest of 13 best methods for evaluating traditional areas of competence in medical
education (18). We next address these methods and
comment on their applicability to psychotherapy. We
TABLE IV
139
EVALUATING COMPETENCE
Patient Surveys
Patients are directly surveyed to assess satisfaction with services provided (e.g., amount of time
spent with the patient, a therapists skill, courtesy,
and empathy or a therapists provision of information
about interview ndings, treatment steps, and/or policies). Specic aspects of psychotherapy, including
overall treatment outcome, working alliance, global
session evaluation, and patient reactions to individual interventions, can also be surveyed (16). Reliability estimates of 0.90 or greater (1.0 is perfect agreement) have been achieved in rating hospitals and
clinics but are much lower for individual trainees
(18). Difculties encountered with patient surveys
are: 1) language and literacy problems; 2) obtaining
enough surveys per resident to provide reproducible
results; 3) the resources required to collect, aggregate,
and report survey responses; and 4) assessment of the
trainees contribution to a patients care separate from
that of the treatment team.
360-Degree Evaluation Instrument
This method requires assessment by multiple
people in a trainees sphere of inuence (e.g., superiors, peers, subordinates, patients and families). To
our knowledge, there are no published reports of 360degree evaluations as applied to graduate medical
education. In business, military, and education settings, reliability has been estimated as high as 0.90. It
is theoretically appealing to collect reliable, primary
Table V.
Beginner
Competent
Procient
Expert
isolated facts
some choice
some synthesis,
self-control
independence,
identity
professional norms,
patient-centered
internalized
lectures, labs,
faculty control
seminars, labs
supervised work
realistic work
setting
specialized training,
socialization
self-managed
tests
portfolios
simulations
markers
real evaluations,
internalized standards
work-related
self-assessment
Learning Issues:
Learning Methods:
Evaluation Method:
140
MANRING ET AL.
Portfolios
A portfolio is a collection of products prepared
by the trainee that provides evidence of learning and
achievement related to training goals. It typically contains written documents, such as logs and transcripts
but can also include video or audio recordings. It can
include statements about what has been learned,
statements about application, and statements about
remaining learning needs and how they can be met.
In general psychiatry training, a portfolio might include a log of diagnoses treated, a log of therapies
used, a summary of the research literature reviewed
in selecting a treatment option, a quality improvement project, ethical dilemmas faced and how they
were handled, and recordings or transcripts of actual
therapy provided to patients. The contents of a portfolio do not have to be standardized since the purpose is to demonstrate individual learning gains relative to individual goals. A portfolio is also one of the
best tools for combining teaching with assessing continuity-of-care concerns that are central to psychotherapy. A portfolio for psychotherapy might include
sample transcripts or tape segments demonstrating
transference, resistance, countertransference, patients dysfunctional thought records or cognitive distortions, vignettes describing relationships between
past experiences and a patients current symptoms,
and examples of nding a focus with patients in brief
therapy. Developing protocols for assessing such
141
EVALUATING COMPETENCE
summarize a trainees breadth of experience they offer little help in assessing skill competence.
Record Review
Simulations used for assessment of clinical performance closely imitate reality. They incorporate a
wide array of options resembling reality, allow examinees to reason through a clinical problem, permit
examinees to make life threatening errors without
hurting a real patient, provide instant feedback so examinees can correct a mistake in action, and can be
used to rate examinees performances on clinical
problems that are difcult to evaluate effectively in
other circumstances. Simulation formats have been
developed as paper and pencil branching problems
(patient management problems or PMPs), computerized versions of PMPs called clinical case simulations (CCX), role playing situations (i.e., standardized
patients (SP)), clinical team simulations, and combinations of all three formats. Scoring rules are set by
experts.
To build a simulation, clinical experts craft scenarios from real patient cases to focus on specic
skills. Technical experts then create scripts for standardized patients or computer based simulations and
add, when feasible, automated scoring rules. Simulations are expensive to create. To our knowledge,
none is currently available for assessing the nuances
of psychotherapy. However, simple clinical vignettes
to assess the recognition and handling of psychotherapeutic phenomena have been incorporated into
the Columbia Psychodynamic Psychotherapy Skills
Test (an MCQ). We expect that the rich literature on
simulation and gaming (1921) will power the development of sophisticated and useful computerized
simulations of the more subtle aspects of therapy in
the future.
While reviews of patient records using standardized protocols may yield information about presenting symptoms, diagnoses, type of therapy performed
and patient outcomes, clinical records do not easily
lend themselves to the assessment of competence in
psychotherapy.
The standardized oral examination uses information from realistic patient cases and a trained examiner, who presents a clinical problem and then
asks the examinee to manage the case. While many
medical specialty boards use standardized oral examinations as the nal examination for initial certication, this method has not been developed to
Academic Psychiatry, 27:3, Fall 2003
MANRING ET AL.
EVALUATING COMPETENCE
chotherapy competence in training programs. Recorded interviews can provide added exibility for
both learning and evaluation. Cross-sectional exams
have limited ability to assess the skills necessary for
competence in the processes of repeated contacts with
a given patient over time central to ongoing psychotherapy.
At present, an achievable psychotherapy assessment program might include 1) sequential tests of
knowledge and understanding using a MCQ-based
exam that incorporates patient vignettes such as employed by the Columbia Psychotherapy Skills Test
and 2) use of a portfolio of recordings or direct observations of psychotherapy sessions in which the
technical skills being assessed are displayed. Ideally,
recordings can be obtained with patients for whom
360-degree ratings or patient surveys show signicant patient improvement as well as patients who
dont improve. In this way trainees may benet from
multiple sources of direct feedback, while evaluators
see the trainees using techniques which actually produce or fail to produce therapeutic change in a specic individual.
As controlled trials inform us about which specic psychotherapies are most effective for which
clinical problems, and which specic behaviors and
techniques are essential for those psychotherapies,
our expectations of trainees will become more rened. Future developments may also foster the construction of sophisticated computer models of psychotherapeutic situations which call on those
essential skills. In the meantime, we will likely be best
served by a broad-based portfolio containing MCQs,
global ratings of ongoing supervision, recorded samples that demonstrate specic technical skills, and nally, by tolerating the ambiguity that has been integral to our profession from its inception.
The authors thank Liz Wilding, Ed.D. for her help in
extracting ideas and transforming them into words.
References
1. ACGME Outcome Project: ACGME General Competencies
Version 1.3 (9.28.99), 2000
2. ACGME Program Requirements for Residency Training in
Psychiatry (effective January, 2001), 2002
3. Lambert J, Bergin AE: The effectiveness of psychotherapy, in
Handbook of Psychotherapy and Behavior Change. Edited by
Bergin AE, Gareld SL. New York, Wiley, 1994, pp 143189
4. Klerman GL, Weissman MM, Markowitz JC, Glick I, Wilner
PJ, Mason B, Shear K: Medication and Psychotherapy, in
Handbook of Psychotherapy and Behavior Change. Edited
by Bergin AE, Gareld SL. New York, Wiley, 1994, pp 734
783
5. Hill CE, Helm JE, Spiegel SB, Tichenor V: Development of a
system for categorizing client reactions to therapist interventions. J Counseling Psychology 1988; 35:2736
6. Gelso C, Carter JA: The relationship in counseling and psychotherapy: components, consequences and theoretical antecedents. Counseling Psychologist 1985; 13:155243
7. Horvath AO, Greenberg LS: Development and validation of
the working alliance. J Consulting Psychology 1989; 36:223
233
8. Moras K, Strupp HH: Pretherapy interpersonal relations, patients alliance and outcome of brief therapy. Arch Genl Psychiatry 1982; 39:405409
9. Foreman SA, Marmar CR: Therapist actions that address initially poor therapeutic alliance in psychotherapy. Am J Psychiatry 1985; 142:922926
10. Kivlighan DM: Relationship between counselor use of intentions and clients perception of working alliance. J Counseling
Psychology 1990; 37:2732
11. Kivlighan DM, Schmitz PJ: Counselor technical activity in
cases with improving working alliances and continuing-poor
working alliance. J Counseling Psychology 1992; 39:3238
144