Evidence Based Assessment / Evidence Based ADHD Assessment
The practice of
Evidence-Based Medicine (EBM) is described as the application of evidence
gained from the scientific and research communities to medical practice. These
EBM practices have long been established for treatment and therapeutic
strategies, but are now being recommended to increase the efficacy of many
diagnostic methods and instruments as well.
In the areas of assessment
and test publishing, we refer to this as EBA, or Evidence Based Assessment.
In the mental health professions, many sources referring to EBA
mean to suggest that a particular treatment, intervention or assessment has
some level of empirical evidence to support it's use. In the medical sense, EBA
is more specific to mean that you have evaluated the scientific literature and
reviewed the published findings of the statistical relationships between a
given condition and an assessment measure, treatment or intervention. This is
commonly recorded as the effect size of the relationship. These published
effect sizes can be converted to useful metrics, such as sensitivity and
specificity, which can then be easily translated into Receiver Operating
Characteristics, (ROC), and Likelihood Ratios, (LR). Using likelihood ratios,
the clinician can build a predictive index by combining the results from
multiple lines of evidence, by far one of the most useful approaches in
assessment and treatment. This is referred to as incremental validity or the
use of multiple measures in combination to add greater predictive power to a
diagnosis and proposed treatment plan.
This approach is important because it allows the clinician to
evaluate these inputs while considering the Prevalence, or base rate of a
condition without adversely inflating the predictive power, as can happen when
we rely only on sensitivity or specificity alone.
Sensitivity and Specificity refer to research comparison groups
where 100% group membership is known before you take any action or make any
determination.
For example, the result of a test with 90 percent accuracy should
predict that you could correctly assign 9 out of 10 subjects with that measure
if you knew they came from one of two groups, those with the condition and
those without.
This is a frequently used approach and can often be misleading in
terms of the true diagnostic predictive power. To reliably assess any given
measure’s true predictive power we have to know the degree of relationship
(effect size) of a condition to its proposed assessment or treatment and we
need to know the actual base rate of that condition within the population in
general. In clinical practice, rarely are we faced with determining membership
when we have 100% percent knowledge that a given subject will belong to one of
two groups. For example, with ADHD a conservative base Rate is 4% ADHD and 96%
Non-ADHD. In contrast when using a tests sensitivity and specificity only one
incorrectly assumes they are applying the same predictive power of 100% group
understanding to a condition where it is actually 4% and 96% respectively. In
the EBA approach used in this example of ADHD where the base rate is 4% known,
that same test with 90% sensitivity and specificity will have a limited effect
on a predictive index of 21% increasing our true odds of correctly diagnosing
based on that result alone from 4% to 25%.
This is a far cry from the widely misunderstood belief that a test with
90% sensitivity and specificity will yield an accuracy rate of 90%. If we add a
second measure with 90% accuracy to the EBA process we would improve our
predictive index to 74%, not 90% times two. 74%, while an improvement may not
be appropriate depending on the impact of diagnosing or treating or doing
nothing depending on the risks associated with each. Clearly we want to be more
informed when the stakes are higher. Adding a third measure of 80-90 %
sensitivity and or specificity to the EBA process will push the predictive
index above 90 % which can more effectively guide the decisions to treat or not
treat, diagnose or not diagnose. Alternatively, using multiple measures with
proven effectiveness in this fashion can also push the predictive index in the
opposite direction, away from a positive diagnosis.
The bottom line is that when assessing conditions with relatively
low base rates you will need multiple measures with strong diagnostic evidence
working in conjunction with each other all telling you the same thing if strong
predictive power is to be obtained.
To be effective we must first
determine the correct base rate, or the initial risk that the patient has ADHD.
The Base Rate is also known
as “Prevalence”, or Pre-Test Probability.
Establishing the proper base
rate is important because it will have a dramatic effect on the Post-Test, or
Posterior Probability, (outcome).
Again, using ADHD as an
example:
Base rate estimates of ADHD
can vary widely depending on the type of settings and methodologies used to
calculate them.
Epidemiological estimates of
ADHD have ranged from 3% to 12% (American Academy of Pediatrics, 2000; American
Psychiatric Association, 1994).
Currently reported base rates
can range from 0.03 to 0.74. The higher estimates are from specialized ADHD
clinics and the lower estimates are from recent epidemiological studies.
Specialized neuropsychology clinics may have higher rates than these, however,
a solid rationale must exist for the use of these extremely high base rates.
Higher clinical base rates makes sense because outpatient psychological and
neuropsychological clinics usually have enriched samples as a result of
referral sources weeding out many of the more unlikely cases in advance.
Whenever possible, the
practitioner should calculate the base rate of ADHD based on experiences and
the past history of their particular setting.
To calculate your actual base
rate for ADHD, you must determine the percentage of patients, referred to you
for ADHD assessment, who are then properly diagnosed as ADHD.
(i.e. 100 patients, 34 are
positive for ADHD = 34% base rate).
(# of Patients identified as ADHD / Total # of Patients referred
for ADHD) * 100.
Ex. 34/100 = 0.34, 0.34 X 100 = 34%
This should to be
recalculated periodically to accurately reflect changes in the referral
population.
The preferred method for
applying these base rates and likelihood ratios is with a Fagan’s nomogram.
The Fagan nomogram is a graphical tool for estimating how much the result on a diagnostic test changes the probability that a patient has a disease (NEJM 1975; 293: 257). A picture of the Fagan nomogram appears below.
To
use this tool, you need to provide your best estimate of the probability of the
condition prior to testing. This is usually related to the prevalence of the
condition, though this may be modified up or down based on the factors
previously described You also need to calculate the likelihood ratio for the
diagnostic test result as previously described.
With this information, draw a line connecting the pre-test probability and the likelihood ratio. Extend this line until it intersects with the post-test probability. The point of intersection is the new estimate of the probability that your patient has this disease.
Here are details on how the graph works and how you could construct a similar graph yourself. The principle is very much similar to a slide rule.
First, the computations involved use odds rather than ratios. If you multiply the pre-test odds by the likelihood ratio, you will get the post-test odds. And since multiplication of two numbers is equivalent to adding their logarithms, we use a log scaling for both the odds and the likelihood ratio.
The official formula is:
Post-test odds = likelihood ratio x pre-test odds
and the Fagan nomogram uses the equivalent formula
log(post-test odds) = log(likelihood ratio) + log(pre-test odds)
So although the labels on the left and right are written in terms of probability, the tick marks are spaced at the log odds. For technical reasons. we have to set the scaling of the log likelihood ratio to 1/2 that of the log odds. We also have to invert the scale for the log pre-test odds.
So if you wanted to construct this graph yourself, simply plot a range of log odds at x=+1. Plot an inverted range of log odds at x=+1. Write labels in terms of probabilities rather than odds. Then plot 1/2 of the log likelihood ratio values at x=0.
A sometimes more useful alternative formula using sensitivity and specificity is:
LR(+) = sensitivity / (1-specificity) for tests designed to rule in the condition
LR(-) = (1-sensitivity) / specificity for tests designed to rule out the condition
Here are a couple examples of how to use the Fagan nomogram.
When you draw the
line, you can see the new Probability has gone
from 4% to 25%. When you draw the
line, you can see the new Probability has gone
from 25% to 74%.
Nomogram for Bayes
theorem" Fagan TJ (1975) New
England Journal of Medicine; 293: 257.
http://www4.umdnj.edu/cswaweb/ccpc_pres/nyceebm01/sld035.htm
http://www.hsl.unc.edu/Services/Tutorials/ebm/index.htm