CRITICAL APPRAISAL

Putu Moda Arsana

FKUB,2017
 What is Critical Appraisal?

Critical appraisal is the process of

systematically examining
research evidence to assess its validity, results,
and relevance before using it to inform a
decision

(Hill and Spittlehouse, 2001)

 Why We have to do it ?

• Research involves gathering of data,

collection of data, and analysis of the data to
produce meaningful information.
• However, many of the research are not in
good quality and many studies are biased and
their results are untrue.
• This can lead us to draw false conclusions
 When should you Critically Appraise?

• Conducting literature reviews for grant proposals

for new project/research
• Evaluating the effectiveness, costs, and benefits
of health programs, Intervention etc.,
• Establishing new innovative in the health
programs
• To set right the lacunae while Implementing
health policies and Public health decision
making
How can You appraise the
evidence ?
 Parameters of the critical appraisal

1. V ( validity ) :
“Are the result of the study valid?”
 The answer : see on the method
1. I ( Importance ) :
“Are the valid result of this study important?”
 The answer : see on the result
2. A ( Applicability ) :
“Can you apply this valid, important evidence about this
study for your patients ?”
 The answer : see on the discussion
 Study design and the critical appraisal

Each study has it’s own parameters :

• Treatment study
• Harm / etiology study
• Diagnosis study
• Prognosis study
• Guidelines study
• Systematic review ( meta analysis ) study
 Critical appraisal of the Treatment Study

Are the results of therapeutic trial valid ?

1.Was the assignment of patients to treatments randomized?

2.Was follow-up of patients sufficiently long and complete?
3.Were all patients analysed in the groups to which they were
randomized?
4.Were patients, clinicians, and study personnel kept “blind” to
treatment?
5.Were the groups treated equally, apart from the experimental
treatment?
6.Were the groups similar at the start of the trial apart from the
experimental therapy?
 Randomization

• A method of selecting a sample (random

sample) from a statistical population in
such a way that every possible sample that
could be selected has a predetermined
probability of being selected.
Was follow-up of patients sufficiently long and
complete?
 Were all patients analyzed in the groups to which
they were randomized?

• Most trials will suffer from some loss to follow-up for various
reasons (including patient withdrawal due to
safety/efficacy/consent, lost contact, etc). However, to avoid
bias and maintain the validity of the initial randomisation, loss to
follow-up should be minimal — generally less than 20% of the
randomised population. However, it should be noted that if few
patients have the outcome of interest, then even small losses
can bias the results

• An intention-to-treat analysis is where the groups are analysed

based on the initial treatment to which they were randomised
(not by treatment eventually administered or whether they
finished the trial!). This type of analysis is generally considered
optimal for most RCTs (with the possible exception of non-
inferiority or equivalence trials)
Were patients, clinicians, and study personnel
kept “blind” to treatment?
• Double-blinding — i.e., both patients and investigators
are unaware of treatment allocation is ideal, and
minimises measurement bias. However, sometimes
this may not be possible, for example, where the
intervention involves a physical treatment or surgery or
where adverse effects of a drug clearly compromise
blinding.
• Single-blinding (where either one of the patients or the
assessors are unaware of treatment allocation) may,
however, be possible and acceptable in these
circumstances, depending on the study design
 Were the groups treated equally, apart from the
experimental treatment?

• Bear in mind that this includes all analyses,

tests, and contact with the study personnel,
as well as any other allowed treatments in the
study
• This may not always be possible/ethical
depending on the nature of the interventions;
however, you should decide whether any
differences in treatment during the study
could affect the results
Were the groups similar at the start of the trial apart
from the experimental therapy?

Two methods of assessing the role of chance

•P-values (Hypothesis Testing)
• use statistical test to examine the ‘null’ hypothesis
• associated with “p values” - if p<0.05 then result is
statistically significant

•Confidence Intervals (Estimation)

• estimates the range of values that is likely to
include the true value
Were the groups similar at the start of the trial apart
from the experimental therapy?

• If the randomisation process worked, then the

groups should be similar at baseline (remember
that the goal of randomisation is that everyone
should have a similar and equal chance of being
in either group)
• The study should specify whether any differences
between groups are statistically significant,
because any significant differences between
groups at the beginning of the study could
potentially affect the results
 Are the valid results of this randomized trial
important ?

• What is the magnitude of the treatment

effect?
• How precise is the estimate of the
treatment effect?
 Are the valid results of this randomized trial
important?

What is the magnitude of the treatment CER, EER, ARR, RRR, HR, RR, OR, NNT, NNH
effect?

How precise is the estimate of the 95% CI, 99% CI

treatment effect?
 Magnitude of treatment effect

Parameter of “ Important “ :
 ARR ( Absolute Risk reduction ) : CER – EER
 RRR ( Relative Risk Reduction ) : ( CER – EER ) / CER
 NNT ( Number Needed to treat ) : 1 / ARR

CER : Control Event Rate

EER : Experimental Event Rate
Note : Use 2 x 2 table to account CER and EER
 SAMPLE CALCULATIONS

Occurrence of diabetic neuropathy Relative risk Absolute risk Number

at 5 years among insulin- reduction (RRR) reduction (ARR) needed to
dependent diabetics in the DCCT treat (NNT)
trial

Usual insulin Intensive insulin CER – EER CER – EER 1/ARR

regimen control regimen CER
event rate (CER) experimental
event rate (EER)

9.6% 2.8% 9.6% – 2.8% 9.6% – 2.8% 1/6.8%

9.6% =15 patients
=71% =6.8%

  95% CI 4.4% to 9.2% 11 to 23

 Estimation of treatment effect

95% confidence interval (95%CI) of ARR and NNT

= ARR ± 1.96 CER x (1-CER) EER x (1-EER)

+
number of control patients number of experimental patients

= ARR ± 1.96 0.96 x 0.904 0.028 x 0.972

+
730 711

= ARR ± 2.4% = 4.4% - 9.2%

NNT = 1/ARR = 1/9.2% - 1/4.4% = 11 – 23

 Can you apply this valid, important evidence about
therapy in caring for your patient?

Do these results apply to our patient?

Is our patient so different from those in the study that
its results cannot apply?
Is the treatment feasible in our setting?
What are our patient’s potential benefits and harms from the therapy?
Method I: f Risk of the outcome in our patient, relative to
patients in the trial.
Expressed as a decimal:______
NNT/f=______/______=______
(NNT for patients like ours)
Method II: 1/(PEERRRR) Our patient’s expected event rate if they
received the control treatment (PEER)
=______
1/(PEERRRR)=1/________=______
(NNT for patients like ours)
Are our patient’s values and preferences satisfied by the regimen and its consequences?

Do we and our patient have a clear assessment of their

values and preferences?
Are they met by this regimen and its consequences?
 Can you apply this valid, important evidence about
a treatment in caring for your patients ?

Do these results apply to your patients ?

1. Is your patients so different from those in the trial that its results can’t
help you ?
2. How great would the potential benefit of therapy actually be for your
individual patient ?
3. Method I : Risk of the outcome in your patient, relative to patients in
the trial ( NNT / F = ………………. )
4. Method II : Your patient’s expected event rate if they received the
control treatment ( 1 / ( PEER x RRR ) = 1 / ……
Are your patient’s values and preferences satisfied by the
regimen and its consequences ?
1. Do your patient and you have a clear assessment of their values and
preferences ?
2. Are they met by this regimen and its consequences ?
CLINICAL SCENARIO (modified)

A 65-year-old man is seen in our office after being

discharged from hospital 2 weeks previously. During this
admission he suffered a transient ischemic attack (TIA),
and being diagnosed with carotid stenosis. His hospital
stay was uncomplicated and his discharged medications
included metoprolol50 mg BID for hypertension and
aspirin 81 mg daily. Today, he brought us an article from
the internet describing the benefits of statinsfor stroke
prevention and he wonders what this medication is and if
he should take it. Our note from his last visit showed that
his TC was 5.0 mmol/L, HDL-C was 2.0 mmol/L, and
LDL-C was 2.0 mmol/L.
His examination was unremarkable

(Strauss SE, et.al.Therapy in: Evidence-Based Medicine, 2005)

 Steps

• Ask a foreground question

• Acquire some articles
• Appraise the evidence
• Apply the findings
• Assess your performance
 To treat or not to treat?

• Validity
• Importance
• Applicability
GOOD CLINICAL TRIAL

• VALID
• IMPORTANT
• APPLICABLE
 Are the study valid?

1.Was the assignment of Patients randomized?

2. Was follow-up sufficiently long & complete?
3. Was all Patients analyzed in groups?
4. Were Patients & Doctor kept blind to Rx?
5.Were groups treated equally?
6. Were groups similar at start of study?
 Are the study valid?

1.Was the assignment of Patients randomized? Yes!!

2. Was follow-up sufficiently long & complete?
3. Was all Patients analyzed in groups?
4. Were Patients & Doctor kept blind to Rx?
5.Were groups treated equally?
6. Were groups similar at start of study?
 Are the study valid?
1. Was the assignment of Pxrandomized? Yes
2. Was follow-up sufficiently long & complete?
3. Was all Pxanalyzed in groups?
4. Were Px& Dr kept blind to Rx?
5. Were groups treated equally?
6. Were groups similar at start of study?
 Are the study valid?
1. Was the assignment of Pxrandomized? Yes
2. Was follow-up sufficiently long & complete? Yes
3. Was all Pxanalyzed in groups?
4. Were Px& Dr kept blind to Rx?
5. Were groups treated equally?
6. Were groups similar at start of study?
 Are the study valid?
1. Was the assignment of Pxrandomized? Yes
2. Was follow-up sufficiently long & complete? Yes
3. Was all Pxanalyzed in groups?
4. Were Px& Dr kept blind to Rx?
5. Were groups treated equally?
6. Were groups similar at start of study?
 Loss of follow up :
15+10=25
25/4731= 0.53%

Intention to treat analysis

3. Was all Patients analyzed in groups?
What is meant by intention to treat analysis? Survey of
published randomized controlled trials

(Hollis S, Campbell F. BMJ 1999; 319: 670 -4)

 Are the study valid?

1.Was the assignment of Pxrandomized? Yes

2. Was follow-up sufficiently long & complete? Yes
3. Was all Px analyzed in groups? Yes
4. Were Px& Dr kept blind to Rx?
5.Were groups treated equally?
6. Were groups similar at start of study?
 Are the study valid?

1.Was the assignment of Pxrandomized? Yes

2. Was follow-up sufficiently long & complete? Yes
3. Was all Pxanalyzed in groups? Yes
4. Were Px& Dr kept blind to Rx?
5.Were groups treated equally?
6. Were groups similar at start of study?
 Are the study valid?

1.Was the assignment of Pxrandomized? Yes

2. Was follow-up sufficiently long & complete? Yes
3. Was all Pxanalyzed in groups? Yes
4. Were Px& Dr kept blind to Rx? Yes
5.Were groups treated equally?
6. Were groups similar at start of study?
 Are the study valid?

1.Was the assignment of Pxrandomized? Yes

2. Was follow-up sufficiently long & complete? Yes
3. Was all Pxanalyzed in groups? Yes
4. Were Px& Dr kept blind to Rx? Yes
5.Were groups treated equally?
6. Were groups similar at start of study?
 Are the study valid?

1.Was the assignment of Pxrandomized? Yes

2. Was follow-up sufficiently long & complete? Yes
3. Was all Pxanalyzed in groups? Yes
4. Were Px& Dr kept blind to Rx? Yes
5.Were groups treated equally? Yes
6. Were groups similar at start of study?
 Are the study valid?

1.Was the assignment of Pxrandomized? Yes

2. Was follow-up sufficiently long & complete? Yes
3. Was all Pxanalyzed in groups? Yes
4. Were Px& Dr kept blind to Rx? Yes
5.Were groups treated equally? Yes
6. Were groups similar at start of study?
 Are the study valid?

1.Was the assignment of Pxrandomized? Yes

2. Was follow-up sufficiently long & complete? Yes
3. Was all Pxanalyzed in groups? Yes
4. Were Px& Dr kept blind to Rx? Yes
5.Were groups treated equally? Yes
6. Were groups similar at start of study? Yes
Is SPARCL Study valid?

• Yes this is valid !

 Are the valid results of
the study important?

1.What is the magnitude of R/ effect?

2.How precise is this estimate of R/ effect?
 IMPORTANCE
1. MAGNITUDE OF THE TREATMENT
• P value
• Relative Risk Reduction (RRR)
• Relative Risk Increase (RRR)
• Absolute Risk Reduction (ARR)
• Absolute Risk Increase (ARI)
• Number Needed to Treat (NNT)
• Number Needed to Harm (NNH)

2. HOW PRECISE IS THE TREAMENT

EFFECT?
• Confidence Interval 95% (CI 95%)
NNT
•Number of patients should be treated to avoid 1 bad
outcome (avoid outcome)
•Number of patients should be treated to
have 1 additional good outcome (have outcome)

NNH
•Number of patients treated to harm one patient from the
therapy
 Absolute Risk Reduction Primary Outcome
on Fatal or Non fatal stroke

• ABSOLUTE RISK REDUCTION (ARR) : 13.1%-11.2% =

1.9 %

• NNT = 1/ARR = 1/1.9% = 100/1.9 = 53

 Absolute Risk Increase (ARI) On Hemorrhagic
Stroke (HS)

• Atorva event rate (HS) : 55/2365 X 100%= 2.32%

• Placebo event rate (HS) : 33/2366 X 100%= 1.4%

• ARI HS : 2.32%-1.4% = 0.92 %

• Number Needed to Harm (NNH) :
• 1/ARI = 1/0.92% = 108

Note: kejadian HS pada atorva ada55 pasien dari 2365 pasien group
atorva. Sedangkan pada plasebo sebanyak 33 dari 2366 pasien group
plasebo
 LHH (likelihood of being help versus harmed) in
Regards Hemorrhagic Stroke vs Primary Outcome
(non fatal or fatal stroke)

LHH = (1/NNT) vs (1/NNH)

1/53 vs 1/108

2.03 times
ARTINYA: kecenderungan untuk memberikan atorvastatin
benefitnya dua kali lebih baik dibandingkan dengan hanya
memberikan plasebo
Is the results of SPARCL study
important?

• Yes this is important!

Is there any higher evidences?