Who
and What to Believe? | Biases | Associations:
Causal
or Coincidental? |
Baseball & the FDA | Billiards and Clinical trials: Retrospective
versus Prospective
Identifying Quackery | Is there an Impartial group?
| Terminology | Types of clinical data (best first) |
The problem with Testimonials | Weighing Sources of Medical Evidence |
References
What to believe?
What makes information credible?
Weighing types of evidence
"Evidence-based medicine is the
conscientious, explicit and judicious use of
current best evidence in
making decisions about the care of individual patients." cebm.net
The importance of trying to see things as they are:
Bobby Jones, a golfing great, when responding to a question about his
crippling disease later in
life replied: "We all have to play the ball as it lies."
His words serve to remind us that we must strive to see our situation - the ball, the wind and grass, the available clubs - as objectively and clearly as possible, and strive to make informed decisions based on the information at hand.
Our goal is to help patients and caregivers better evaluate medical claims, to:
Help distinguish between strong and weak information
Learn to avoid confusing associations with causality
Learn how to better recognize information that is not plausible or is weakly supported
How do we start?
By questioning (not just accepting) the information we receive and by recognizing factors that can distort medical claims and scientific findings, such as conflicts of interest and biases; and by understanding the scientific methods used to evaluate therapies for cancer and other medical conditions.
Bias: different reasons for
different groups:
"Bias" has two meaning:
(1) "an inclination of temperament or outlook; especially: a personal and sometimes unreasoned judgment." Bias can influence how one looks at outcomes, or in what we choose to read or ignore.
(2) In a study design, a bias is defined as an error in the method of study that leads to a deviation in the outcome away from the truth.
Sources of Bias:
Financial
conflict of interest
Conflicts of interest occur when personal, professional, or financial interests intentionally or unintentionally influence decisions on: 1) scientific methods, or 2) how data from the study are interpreted.
"A financial conflict of interest, I believe, is any financial association that would cause an investigator to prefer one outcome of his research to another. Let me give you an example. If an investigator is comparing drug A with drug B and owns a large amount of stock in the company that makes drug A, he will prefer to find that drug A is better than drug B. That is a conflict of interest."
~ Marcia Angell, M. D. Source: ohrp.osophs.dhhs.gov
Those who develop new drugs or sell supplements have an inherent financial conflict of interest with respect to objectively evaluating the true worth and benefits of their products or services, which can lead to selective reporting or "hyping" the products in order to maximize shareholder confidence and the profitability of the company that sells them.
Scientists who have financial interests in products or services must disclose these relationships, which make them inherently less able to overcome the biases these monetary interests can create.
"While most people think conflicts of
interest are a problem of overt corruption, that is, that
professionals consciously and intentionally misrepresent the
advice they give so as to secure personal gain, considerable
research suggests that bias is more frequently the result of
motivational processes that are unintentional and unconscious
" 3
Consider that, by investing in a company, a scientist may demonstrate a belief in the value of its product, perhaps in advance of evidence. Scientists have an ethical responsibility to avoid arriving at conclusions ahead of time. The discipline of good science requires that theories be tested in well-controlled studies, and that the outcomes be evaluated objectively before conclusions are made.
ASK:
"Was
as the research funded by an organization that generally advocates a specific point of view?
Do the findings of the research parallel the organization's point of
view - or too closely?"
2
Reporting bias: Because of personal inclinations, expectations, or other biases, a participant in a study may report events more favorable to the hypothesis, and leave out, or not see, events that contradict it. There is also the potential for this type of bias in case reports.
Selection bias: Investigators may introduce bias into a study by selecting patients who have characteristics (such as young age) that are favorable to a desired outcome; and also by excluding patients who do not (such as patients with low blood counts). Randomized studies protect against this kind of bias.
ASK:
Was the data source manipulated to produce analyzable data?
Was the data source selected with an unusual methodology?
Was the data source unusually small or narrow, or tightly controlled by the researcher?
Was the data source self-selected (not random)?
Intellectual/confirmation
bias: Investigators and scientists can develop unintended prejudices about the value of their work, ideas, or intellectual property.
Investigators may unconsciously see benefit when none exists, or they may set up a study in ways more likely to reveal weaknesses, exaggerate the benefits, overlook unanticipated side effects, and so on.
"I
know that most men, including those at ease with problems of the greatest
complexity, can seldom accept even the simplest and most obvious truth if it
be such as would oblige them to admit the falsity of conclusions which they
have delighted in explaining to colleagues, which they have proudly taught
to others, and which they have woven, thread by thread, into the fabric of
their lives.”
~ Leo Tolstoy
ASK:
Is the researcher affiliated with an organization that promotes a specific point of view?
Does the researcher produce studies that consistently generate the same conclusions?
2
"Is the research group the only group interested in the
research question?
Confirmation
bias is the tendency to give more weight to
incidents and data that conform to preexisting beliefs and to forget
things that do not.
"We're all prone to it,
including scientists. One major advantage of the scientific method is
that it is pretty good at overcoming confirmation bias."
Source: http://scienceblogs.com
Resources:
Biases common to patients and caregivers
Denial: In order to cope with living with a life-threatening disease, patients or caregivers may develop a tendency to minimize the dangers of the disease, or to inflate the potential of alternative and other less toxic approaches to control it. Denial can
lead to missed opportunities and delays that can make the disease more difficult to treat.
Fear: To be fearful of a cancer or cancer treatment is to be human, and sometimes it's justified. In patients, the fear of the toxicity associated with many standard cancer therapies can form a bias in favor of claims made for safer alternative, or even investigational low-toxic therapies.
Physician biases (reasons to
consult independent experts):
Even a trained oncologist can have conflicts of interest, biases, or gaps in knowledge - especially if he or she does not specialize in lymphomas.
Investigators may have intellectual biases about any therapies they may be testing.
Community doctors might have biases in favor of what is easiest to administer.
HMO physicians may prescribe what is least expensive. Other doctors might be influenced, perhaps unconsciously, by sales promotions from the drug industry.
Patients expressing their desire to continue working without interruption may influence a busy physician to prescribe what meets the immediate needs, without fully discussing possible negative long-term implications of that treatment decision.
NOTE:
(The above section was improved by Kathy aka
gracie82159 on 3/7/2008)
References and Further Reading:
For a readable and concise paper on scientific integrity
ethics.ucsd.edu
Potential
Sources of Bias ascd.org
pdf
The Dirt on Coming Clean: Perverse Effects of
Disclosing
Conflicts of Interest cbdr.cmu.edu
.pdf
-
Identifying Your Biases: wpas-rights.org
"It is important to recognize your biases, and to periodically
evaluate whether they are interfering with your judgment. The first
step to not letting bias interfere with your judgment is to accept that it's
there and decide to deal with it."
Associations:
Did an action lead
to an outcome - a change for good or ill?
Was the effect causal or coincidental?
"It was found by "rigorous" analysis of medical records that
vaccination leads to air travel." ~ unknown
It's natural to look for causal connections
between one event and another, but it's important to consider that
chance or numerous other factors could also explain the
"effect."
An
association is an observation that one event or condition occurs
with another. But associations do not mean
one thing caused another. That is, associations do not prove causality
- the relating of causes to the effects they produce.
Example: A study finds that people who drink wine are healthier than those who drink beer.
But is it the wine? It may be that people who choose wines are more likely to eat healthier foods (culturally?), or that foods that go well with wine are better for you than foods that go well with beer
(the chips and pizza are confounding variables).
Thus, consumption of wine might not influence health one way or the other.
Forming conclusions that A caused
B
can range from harmless to dangerous:
Harmless, when a baseball player hits a home run and recalls
that he had eggs for breakfast that day - who then continues to have
eggs on game days;
Dangerous, when a patient delays a needed treatment based on
anecdotal reports (an association taken as fact) that use of an herb can control the disease.
Making observations and exploring *potential* connections between events (forming hypothesis) is the *starting point* of science
- inspiring further research and well-controlled experiments that prove or disprove the observed
associations.
Importantly, it can sometimes be harmful to form beliefs based on
anecdotes, and testimonials, even when supported by
what appear to be very logical ideas or preclinical science.
Here are some common assumptions about cause and effect related to indolent
lymphoma:
 | My disease is stable, therefore the
life style changes I have made are helping.1 |
|
A drug
resistance assay predicted my response to treatment and my
response was great.
Therefore, the assay
is proven to be valid. 2 |
| My lymph nodes are regressing,
therefore the investigational vaccine I took is effective.2 |
| My lymph nodes started growing
after I had chocolates, therefore chocolate cause progression.1 |
| My lymph nodes started regressing
while taking Aloe Vera, therefore it's effective against the
disease.1 |
1 Patients with indolent
lymphomas may be particularly susceptible to
confusing cause and effect, because the natural course of the disease
is so variable.
For example, it may remain stable for many years
without any intervention, or regress
spontaneously "as many as 20% to 30% of patients will experience regressions at
some time in the clinical course of their disease."
Therefore, if a practitioner prescribes a
life style or alternative protocol that 100 patients follow, as many as 30%
are likely to do well
because they would have done well anyway. This "effect," - which has good
probability of being unrelated to the practice - will often result in strong
belief and promotions, as in: "How can you argue with success?"
2 Lymphoma is sensitive to many
treatments. The proof that an assay can predict response will
require controlled studies on many patients over time.
Providing patient testimonials is not a substitute for well designed
studies.
This is not to say some unproven
practices are not helpful in some way or
degree. And if the practices are harmless and based on
plausible theory we should not condemn them out of hand, so long as
the patients are informed of the alternative explanations.
In other
words, we believe that practitioners have an obligation to state that their
ideas and practices have not been proven to provide benefit.
Furthermore, patients should be aware of potential intellectual
or financial biases - such as does the individual profit from me using his or her unorthodox protocol?
Part to whole:
John Godfrey Saxe's version of an
Indian parable provides us solace
as we gradually move from what we
think to what we know:
It was six men of Indostan
To learning much inclined,
Who went to see the Elephant
(Though all of them were blind),
That each by observation
Might satisfy his mind.
The First approached the Elephant,
And happening to fall
Against his broad and sturdy side,
At once began to bawl:
"God bless me! but the Elephant
Is very like a wall!"
The Second, feeling of the tusk
Cried, "Ho! what have we here,
So very round and smooth and sharp?
To me 'tis mighty clear
This wonder of an Elephant
Is very like a spear!"
The Third approached the animal,
And happening to take
The squirming trunk within his hands,
Thus boldly up he spake:
"I see," quoth he, "the Elephant
Is very like a snake!"
The Fourth reached out an eager hand,
And felt about the knee:
"What most this wondrous beast is like
Is mighty plain," quoth he;
"Tis clear enough the Elephant
Is very like a tree!"
The Fifth, who chanced to touch the ear,
Said: "E'en the blindest man
Can tell what this resembles most;
Deny the fact who can,
This marvel of an Elephant
Is very like a fan!"
The Sixth no sooner had begun
About the beast to grope,
Than, seizing on the swinging tail
That fell within his scope.
"I see," quoth he, "the Elephant
Is very like a rope!"
And so these men of Indostan
Disputed loud and long,
Each in his own opinion
Exceeding stiff and strong,
Though each was partly in the right,
And all were in the wrong!
Baseball and the FDA:
Pitchers often believe every close pitch is a strike and
that the strike zone is too small. So just as we recognize the need
for a neutral party to call balls and strikes in baseball, we
should also recognize the more urgent need for an impartial agency, such as the FDA, to
evaluate claims of medical benefits and risks.
In order to test claims the drug sponsor must conduct
well-designed studies that minimize bias and demonstrate safety and efficacy
for a given condition. Absent evidence-based tests and assessments we'd
return to a "Wild West"
environment with no means of making informed or safe medical decisions, and
no good foundation on which advances in clinical science could be made.
Is the FDA without bias and conflict of interest?
No agency or human
being is completely free of bias, but the agency is committed and mandated
by law to achieve impartiality. There are strong policies on ethics and
conflicts of interest in place, and criminal penalties can be invoked for
violations of these regulations.
About FDA's Ethics
Program: "The
Agency’s ethics program is administered to help ensure that decisions made
by Agency employees are not, nor appear to be, tainted by any question of
conflict of interest. The "ethics" laws and regulations were
established to promote and strengthen the public’s confidence in the
integrity of the Federal Government. The Agency’s Ethics and Integrity
Branch provides advice and assistance to FDA employees on a variety of
ethics related matters including, but not limited to, financial disclosure,
prohibited financial interests, outside activities, co-sponsorship
agreements, and post employment."
See FDA.gov/opacom/ethics/
Billiards and Clinical Trial Design:
Retrospective versus
Prospective trial designs
There are two basic types of studies: retrospective and prospective.
Understanding each can help the patient advocate to better converse with scientists on drug development and assessment ... and help consumers (all of us) to evaluate the strength of evidence in scientific reports, and other medical claims.
A patient advocate provided a nice analogy to help compare retrospective and prospective studies:
"Shoot a cue ball into a pack of billiard balls and the 7 ball goes in the side pocket. A retrospective analysis *looks back* at the shot with the objective of finding evidence that guides how to play pool to win. The prospective study, on the other hand, starts with a hypothesis and tests it going forward: "I will shoot the ball into the pack this way, and I predict the 7 ball will go in the side pocket." So with a prospective study you must
call your shot in advance.
Thus, the prospective study provides a much higher level of confidence that the outcome was determined by the action (that is was causal), and can be repeated ... that it was not the result of chance or other factors.
Is the results of a single prospective experiment sufficient evidence? Generally not, unless the findings are "robust." Most often a second experiment will be needed to validate the first. You also want to scrutinize the DESIGN of the experiment to see if it
contains BIASES (study flaws) that may have "rigged" the outcome. ... Perhaps the 7-ball was put near the side pocket, or the table slants that way.
An example of bias in a clinical trial is when investigators select "ideal" patients that have a favorable prognosis (good counts, young age) ... or they don't count participants in the analysis who died from "unrelated" causes, or they do not give sufficient weight to side
effects ...
The main purpose of doing controlled experiments is to achieve an acceptable LEVEL OF CONFIDENCE that the positive effects measured in the experiment predicts what will happen to patients in the real world. The alternative to this expensive process is to rely on OPINION ... a return to the dark ages.
Importantly, there is never absolute certainty in these matters. Statistics is about measuring the level of certainty that an outcome in an experiment
predicts outcomes for the rest of us ... so that we can have confidence that making a new drug available for an indication (cancer, diabetes, osteoporosis) is on balance better for the patients afflicted with the disease than no treatment, or an existing treatment.
So the billiard table analogy is useful but it oversimplifies. A clinical trial is many times more complicated. For example: what is the outcome that you are measuring (the end point), and how well does it
predict clinical benefit? Does tumor shrinkage increase survival, or outweigh the risk
bone marrow toxicity? Does an increase in time to progression offset the long-term risk of secondary MDS? Does the intervention improve overall survival or quality of life? Thus, the indication (cancer versus a cold), and what's already available to treat it, has a lot to do with how much risk is acceptable for the new drug.
Not surprisingly the drug sponsor will have a bias, because they are driven to do this difficult work in order to realize a profit. So the industry is prone to setting
up or interpreting the experiment in ways that favor the benefit side of the equation.
Please note, however, that the PROFIT motive is ESSENTIAL to the process and to progress Without it new drugs would NEVER be developed or tested. We need all of
these: the profit incentive, rigorous scientific method (controlled prospective studies), patient participation, and independent
and impartial FDA review.
The barometer that we are heading in the right direction in general is an increase in life expectancy, and overall survival (OS) for various indications: Note the recent improvements in OS for indolent
follicular NHL.
It's worth emphasizing that the purpose of conducting well-design trials is to avoid the many dangers of practicing medicine based on opinion. We want to be sure that an intervention - for a specific condition - provides clinical benefit. ...
For example, without use of a controlled study, Hormone Replacement Therapy (HRT) would still be a common practice today ... and, contrary to what was anticipated based on case observation and theories, we would still be giving these hormones to women despite the fact that
HRT significantly increases the risk of heart disease and cancers.
Finally, importantly, with opinion-based medicine we would have no scientific foundation to build on. With the value of drug A based on opinion (observations and theory), we could not reliably compare it to drug B, or evaluate how prudent it is to test drug C with drug A.
Take away standards for approval -- as the Abigail Alliance appears to
advocate for -- we would soon be victimized by claims, counterclaims, sales pitches, and promotions of
inadequately tested drugs. Drugs would be released into the market with insufficient safety and efficacy information. Gathering this information is very much more difficult
after marketing - and that absent standards of evidence for the release of new drugs, the difficulty would increase exponentially.
~ Karl Schwartz
Is anyone's perspective
inherently more believable?
Perhaps we can learn quickly from the
perspectives of doctors and
scientists who have cancer, as the threatening nature of the disease
is likely to remove any financial biases they may have had in
respect to the integrity of the drug evaluation system in America.
And if a secret alternative cures exists, won't these professionals
have the background and the strong motivation to recognize and
validate these approaches?
"Patients who don’t understand the difference
between information based on theory, anecdote, historical
analysis, or double-blind placebo controlled studies are making
ill-informed decisions, believing alternative therapies are safer
or more effective when they are not.
Even patients who presume
that alternative therapies are ineffective may use them. Why?
When faced with a life-threatening disease requiring highly toxic
treatments with no guarantees, or when dying because there are no
effective conventional treatments, it takes guts to reject
something or someone claiming to be able to save you, just in
case you might be wrong." - Wendy S.
Harpham, MD (NHL survivor) Full text: amcancersoc.org
Conspiracy theories are
often used to promote unorthodox therapies
Please consider that such a
conspiracy would require the complicity of many thousands of scientists,
doctors, and regulators - who also get cancer and
spouses, parents, grandparents, children, and loved ones also get
cancer.
Quackery and Charlatanism:
| How Quackery Sells - quackwatch.org
"Most people think that quackery is easy to spot. Often it
is not. Its promoters wear the cloak of science."
|
| On Charlatans and Quackery - grg.org
"Drs. Barrett and Herbert define a quack as anyone who
fraudulently pretends to medical skills they do not possess. They
distinguish among three types: dumb quacks (ignorant), deluded
quacks (self-righteous, true believers), and lastly dishonest
quacks "
|
| What is Pseudoscience? - sciencethinking.com
The term pseudoscience can be applied to any information masquerading
as science. The fakery may be obvious, as in the case of supermarket
tabloids, or much more subtle, and potentially harmful, as is the case
when well-known personalities recommend unproven remedies for serious
medical conditions.
What are some signs that can alert us to the presence of
pseudoscience?
|
| Myth debunked: Traditional Medicine: Identifying Potential Cancer Treatments Of Herbal Origin
http://www.sciencedaily.com
"ScienceDaily (Mar. 5, 2008) — Curing cancer with natural products -- a case for shamans and herbalists?
Not at all, for many chemotherapies to fight cancer applied in modern medicine are natural products or were developed on the basis of natural substances. Thus, taxanes used in prostate and breast cancer treatment are made from yew trees.
The popular periwinkle plant, which grows along the ground of many front yards, is the source of vinca alkaloids that are effective, for example, against malignant lymphomas.
The modern anti-cancer drugs topotecan and irinotecan are derived from a constituent of the Chinese Happy Tree."
|
Terminology to help you assess clinical data and
medical claims:
"All scientific work is incomplete - whether
it be observational or experimental.
All scientific work is liable to be upset or modified by advancing
knowledge.
That does not confer upon us a freedom to ignore the knowledge
we already have or postpone the action that it appears to demand at a
given time. "
- Sir Austin Bradford Hill (1965)
Abstracts
| Theory | Treatment
Response | Statistical Significance
(p-value & confidence)
| Abstracts:
Abstracts are summaries of larger papers and therefore do not
contain all the available details of the study methods and
data.
Abstract conclusions may not be accepted by experts in
the field. Reputable peer-reviewed journals sometimes require
modifications to conclusions from the original abstract for this
reason -- or they may reject the paper from publication because it
was determined that the methods (methodology) or data did not
support the conclusions made in the abstract.
Therefore, it's
important to avoid forming conclusions on the basis of
abstracts. They should be considered only a starting point
for discussions with your doctors and perhaps a basis for
additional inquires and research. |
Linda writes: "I think
it's important to note that one must be very cautious in drawing conclusions from merely reading an abstract.
It's important to read the full article, understand the methodology,
and the strength of the statistics and research design to determine if the conclusions the authors present in the abstract are reasonable.
The better the journal
-- and the higher the quality of the peer review necessary to be published in the journal, the more likely
the methods and design, etc. will be good. Even with that, I've seen some questionable studies get into good journals."
- L - (NHL-survivor & medical professional)
Questions to ask:
1) Was the
paper published in a respected journal?
2) What types of studies and
methods were used to reach the conclusions?
3) Do papers published
by other groups support the conclusions?
Reproducibility is valued in science, especially when a finding
comes from a different investigative group. The reason for this is
that it reduces the chance that bias, or choice of methods, or
chance influenced the findings
or the conclusions.
Key: start by asking questions of (not just accepting) the information we
receive.
|
A theory
is an idea or hypothesis.
"A hypothesis consists either of a suggested explanation for a phenomenon
(observable event or, quite literally, something that can be seen.)" wikipedia.org
Theories are starting points for experiments and studies.
They should not be regarded as a proof. When someone tells
you that this is how a treatment works and that it is therefore desirable,
you might ask:
What clinical data
supports the theory?
Who published the findings, and in what
journal?
Evidence-based medicine requires that a theory - the hypothesis - be tested
objectively ...
in a way that minimizes biases - that the pool table is not tilted in a
way that favors the theory.
|
| Treatment
response describes clinical outcomes from a treatment
based on a clinical change, such as the reduction in size of a lymph
node.
Treatment responses, however, may or may not result in
clinical benefit - improved survival or the reduction of symptoms. For
example, with lymphatic cancers the lymph nodes can increase and decrease
in size because of transient inflammatory reactions, which could lead to false assumptions about the benefit of a
drug or a life style
intervention. Also, the reduction in tumor size might be
offset by the short or long-term toxicities of the drug. |
Some questions
to ask about response:
1) How long was the response?
2) At what
intervals were the outcomes measured and with what tests?
3) Did the
measured response correspond to clinical benefits?
4) Who reported the
responses and were the outcomes verified by independent reviewers?
5) How large was the patient sample,
and how were they selected?
6) What is the expected clinical
course of the disease?
7) What were the short or long-term toxicities (the costs) relative
to other treatments?
| About
statistical significance: In order to draw better
conclusions about data we need to know just a little about
measures of statistical significance, which I think of as a level
of certainty that an outcome was not due to chance, such as
patients with better prognosis in one arm of the study.
So, hypothetically, if you gave two groups identical therapies there would be a difference in the outcome - even if those two groups were quite large. That is, by
chance one or the other group will do better given identical drugs. So it's very common to see what looks to be a sizable difference in
a study outcome be described as "no significant difference."
|
"Statistical conclusions, about responses to treatment in a small
group of patients for example, are not absolute. Everything
is possible, but some things are very possible, some are less
possible and others are very unlikely -- but still possible to
occur.
We draw conclusions with a certain amount of confidence,
conventionally 95% or 99%,
but there is still some chance of an
error (5% or lower)." Source: stanford.edu
We might just look for two
measures of statistical significance in scientific reports to
quickly estimate the strength of the finding: p-value
and confidence interval.
If the p-value is .05 or less, the findings are considered statistically
significant - not due to chance.
The confidence interval (CI) shows the level of confidence
that the study outcome predicts the result in a larger population,
expressed as a range. The wider the range, the less confidence
we have that the results of the study predicts outcomes in the real
world.
Measures of statistical significance
|
Value or range that indicates
statistical significance
|
threshold probability value that tells you if outcome
is due to chance
|
.05 or less
.01 is very good
.05 is on threshold (borderline)
.08 is not statistically significant
|
95% confidence interval (were the study repeated multiple times, it would contain the true effect 95% of the
time) A range of expected results.
|
71% (95% CI: 42-92%)
83% (95% CI: 63-95%)
94% (95% CI: 63-97%)
In the last example, we might say that we are 95% confident that
the response rate is between 63 and 97%.
The wider the confidence interval the less precise the reported value or point estimate.
|
|
P-value
or Probability value: A
value that results from a calculation that tells you how
likely or unlikely the finding (of a difference in treatment
response as an example) was due to chance.
Common language for low p-value: Statistically significant -
means unlikely due to
chance.
Caveats about importance of
P-value:
|
Threshold (cap) is arbitrary, therefore, the closer the p-value is to the threshold (.05), the less statistically
significant.
|
|
Statistical significance does not
necessarily = clinical
importance.
|
|
Chance is rarely the most pressing issue. Biggest threat is systemic error (bias) Therefore more qualitative questions include: Are these the right
patients? Are these the right outcomes? Are there measurement biases? Are observed associations confounded by other
factors?
|
|
P values provide no information on the results'
precision - that is, the degree to which they would vary if measured multiple times. Consequently, journals are increasingly emphasizing a second approach: reporting a range of plausible results, better known as the 95% confidence interval (CI). |
Factors that influence P-values:
|
Magnitude of the main effect: a larger difference will have a lower p-value·
|
|
Number of observations: a difference noted in a study of 500 patients will have a lower p-value than the same differences observed in a 25 pt group.
|
|
Spread of the data (standard deviation): if the observed differences in response are unified and less spread out, the p-value will be lower (more statistically
significant)
Source: Adapted from American College of Physicians-American Society of Internal Medicine |
|
| Formal definition of a 95%
Confidence Interval (CI): "The interval computed from the sample data which, were the study repeated multiple times, would contain the true effect 95% of the time."
" Where confidence intervals are wide, they indicate less
precise estimates of effect." - med.ualberta.ca
Common language for
Confidence Interval:
An estimate of important parameters - how precise or "stable" is your estimate.
|
CI is often reported with a coverage probability of 95% |
|
Confidence intervals are more informative than the simple results of hypothesis tests
since they provide a range of plausible values for the unknown
parameter. |
The wider the confidence interval the less precise the reported value or point estimate.
Simplified
example:
Response rate are:
|
New Drug -- 40/50 pts respond (80% respond) |
|
Standard Drug - 20/40 pts respond (50% respond)
|
-
The difference in response rate is 80-50, or 30%
-
This resulting 95% CI of 30% calculates to: 11%-49%
Since both values in range are above zero, the treatments are significantly
different. We might then say that we are 95%
confident that the difference in response rate between the
new and the standard drug is between 11 and 49%.
How do you calculate the CI?
|
Also see:
|
Difference between p-value and confidence
interval: - musc.edu/
|
|
Three measures of
Association
(adapted from ASSOCIATIONS
& CAUSATIONS http://www.sunmed.org/caus.html
)
risk difference | risk
ratio | odds ratio
Risk Difference (excess risk):
The difference in the incidence among those who are exposed and
those who are unexposed.
EXAMPLE:
If the incidence of lung cancer among smokers is 10 per
1000 and incidence among non-smokers is 1 per 1000,
then the risk difference is 10 per 1000, minus 1
per 1000 = 9
per 1000.
Risk Ratio (Relative Risk or RR):
The ratio of incidence among exposed and unexposed.
EXAMPLE:
using the same smoking example above, this will be 10 per
1000 (smokers),
divided by 1 per 1000 (non-smokers) = 10/1
or 10. An Relative Risk of 10 means that smokers have a
10 times higher risk of
developing lung cancer compared to non-smokers.
A risk ratio is an extremely powerful measure of association.
The greater the RR the more strength you have for the observed
association (in other words, an RR of 10 implies
a much stronger association between smoking and lung cancer compared to an RR of 2).
An RR of 1 implies no association between two variables. In
practice, it's difficult to estimate the RR because true
incidence figures are obtained only from studies which have a
longitudinal component (cohort study) and such studies are
difficult to do.
Hazard Ratio is a kind of relative risk
When comparing groups the hazard ratio is often quoted.
This is the ratio of the risk of an event in one group
relative to the other group at a particular moment. sin-italy.org
It's been described as an estimate of relative
risk, which is a ratio
of the probability
of the event occurring in the exposed group versus the control
(non-exposed) group.
"Hazard ratios have also been used to describe the
outcome of therapeutic trials where the question is
to what extent treatment can shorten the duration
of the illness."
Details (technical): http://aac.asm.org/cgi/content/full/48/8/2787
Odds Ratio or OR (Relative Odds):
The odds (not risk) of occurrence of an event or disease
compared between two groups (exposed and unexposed).
An OR is usually computed in a case control
study where it is not possible to get the true risk
(incidence). Risk and odds tend to very similar when the
disease occurrence is rare'.
EXAMPLE: If a group
finds that lymphatic tissue from patients with NHL are 5 times
more likely than controls (normal tissue) to have a certain virus,
that would be expressed as
OR = 5. An OR of 1 would mean no difference between the two
groups.
|
Absolute risk / Life time risk
The risk of developing a disease over a period of time. We all have absolute risks of developing various diseases such as heart disease, cancer, stroke, etc. The same absolute risk can be expressed in different ways. For example,
you have a 1 in 50 risk of developing a lymphoma in your life. This can
be expressed as a 2% risk, or a 0.02 risk.
Types of Clinical Data (most reliable first)
|
Randomized controlled clinical
trials: (Provides strongest evidence of clinical benefit)
Participants are assigned randomly (by chance) to separate groups (arms)
for the comparison of different treatments -- usually a standard and an
investigational treatment. Patient informed consent is required. Neither
the investigators nor the patient choose the group in which participants
will be placed.
Using chance to assign people to treatment arms helps to avoid selection bias -- putting
pts in better health in the investigational arm, for example. It also
helps to ensure that the groups will be similar and that the treatments
they receive can be compared objectively.
Randomized trials can be "double-blinded"
or "non-blinded." In double-blind studies, neither the
investigator nor the participants are informed of which arm the
participants have been assigned to. This also reduces bias and
improves confidence in the findings.
NOTE: Systematic reviews that evaluate the outcomes in many
trials, including randomized trials, may be the best source of evidence
to guide clinical practice.
|
| Nonrandomized controlled clinical
trials (experimental studies):
Participants are assigned to a treatment group based on criteria
determined by the investigators, such as prognostic indicators,
and disease type. This study design makes it possible for
investigator bias to influence the findings, and therefore there
is less confidence that the group receiving the treatment under
study and the control group are comparable. |
Prospective versus Retrospective studies
"A prospective study watches for outcomes, such as the development of a disease,
during the study period and relates this to other factors such as suspected risk or protection factor(s). The study usually involves taking a cohort of subjects and watching them over a long period. The outcome of interest should be common; otherwise, the number of outcomes observed will be too small to be statistically meaningful (indistinguishable from those that may have arisen by chance). All efforts should be made to avoid sources of bias such as the loss of individuals to follow up during the study. Prospective studies usually have fewer potential sources of bias and confounding than retrospective studies."
"A retrospective study looks backwards and examines exposures to suspected risk or protection factors in relation to an outcome that is established at the start of the study. Many valuable case-control studies, such as Lane and Claypon's 1926 investigation of risk factors for breast cancer, were retrospective investigations. Most sources of error due to confounding and bias are more common in retrospective studies than in prospective studies.
If the outcome of interest is uncommon, however, the size of prospective investigation required to estimate relative risk is often too large to be feasible. In retrospective studies the odds ratio provides an estimate of relative risk. You should take special care to avoid sources of bias and confounding in retrospective studies."
Source: statsdirect.com
| Case series
(observational
studies):
Case series are studies (usually retrospective) that describe outcomes, such as responses, time to
progression, etc.) from patients who received the treatment under
investigation.
These provide weaker evidence than do experimental studies because of the
potential for biases such as, but not limited to, who is observed and
what outcomes the observer is looking for, unknown association
between factors and outcomes -- such as not accounting for other reasons
that could explain the observed result.
The value
of these types of studies (e.g., case series, ecologic, case-control,
cohort) is that they provide preliminary evidence that can be used as
the basis for hypotheses testing in stronger experimental studies, such as
randomized controlled trials. Consider
the recent HRT report finding that using estrogens increases the risk of
heart disease and cancers. The hypothesis that it might reduce these
risks was based on observations that were proven to be incorrect.
|
| About press releases,
and the reports you don't see:
Protocols, ethical principles, and a desire to maintain
credibility are beneficial forces that encourage responsible
public reporting of drug development research and clinical trial
outcomes. But consider that an easy way to put a positive
spin on a company's drug development project is to selectively
report on favorable outcomes, and to keep
less than stellar results from being released at all.
Ask: How is response to treatment being defined? How were the
patients selected? How many patients were tested? What
was the control? Has the finding been replicated by an
independent group? |
| Testimonials (Least reliable)
Patient stories
can be excellent resources. Each tells us something about the range of what's possible, and may therefore provide encouragement, and sometimes important lessons.
We provide a Patient Stories page, which improves with each addition.
http://www.lymphomation.org/stories.htm
Testimonials on the other hand are stories put forth as *evidence* that a choice led to an outcome. The intent is to persuade and promote a medical practice, often one that is not yet proven: "I did this, and I benefited. If you do this you can benefit too." The presumption is that the person knows that his
or her outcome was influenced by the action, and that it predicts what your
experience will be too.
Testimonials cannot inform
about:
 | The number of persons who used an intervention and did not benefit, or
were harmed.
(also applies to case reports)
Compare with peer-review clinical trial where the number of
patients receiving the treatment
are known, and the positive and negative outcomes are measured
uniformly - and reviewed independently.
Patients who have tried and failed an alternative strategy will
not be around to testify. |
| The authenticity of the report, and it's accuracy.
Can we know if the
person reporting the benefit really has the condition,
or is
reporting the outcome accurately? |
| The biases
of the individual reporting his case as evidence.
(also applies to case reports)
Do they have a financial conflict of interest? Do they sell
the product or charge a fee for dispensing the information?
Is the testimonial a way of validating their personal
decision process and theories?
|
| The specifics of the
case
(also applies to case reports)
What is the natural course of the disease? Can
it wax and wane without intervention? Did the intervention cause the
outcome, or was it coincidental?
Even for cancers with a very poor
prognosis there are case reports in the literature of spontaneous remissions, independent of any intervention.
People sometimes win the
lottery , but this does not make playing the lottery a good bet - particularly when betting your life. |
| How the
outcomes were measured, and by whom
(also applies to case reports)
Was it an objectively measured response, or a patient reported
outcome? Was it that the patient felt better?
What happened later? ... did the intervention lead to a lasting clinical benefit? |
| What other medical treatments were given shortly before or after?
A CT scan will often show lesions after standard treatment that are
necrotic scar tissue.
Credit might be given to an alternative practice used after this treatment,
when it was merely
the resolution of a scar tissue, a normal bodily process. |
| The accuracy of the
diagnosis
Was it a false diagnosis of a cancer? |
For all of these reasons, it's prudent to regard testimonials
- stories presented as evidence - with
suspicion.
Similarly, case reports cannot be considered evidence of benefit
or safety.
|
| In-vivo
or in-vitro? - in-vitro means in test tube or cell
culture; in-vivo means in the body.
We frequently read or hear about the anti-cancer properties of
this or that supplement based on scientific research
findings. Here are some questions to ask of this kind of
information:
|
Was the response detected in a test tube
(in-vitro/cell culture)?
The human body is infinitely
more complex than a test tube.
The tumor cells change when removed from the body;
oftentimes they will die spontaneously.
Nevertheless, indications
of activity in a test tube often become the basis for product claims about
natural supplements. Be aware that this can only be
a starting point for additional experiments. Using such
data as the basis of medical claims is irresponsible, and
bias should be suspected. Furthermore, you might ask
if the dose used to produce the in-vitro
effect possible to achieve in the body, or if it can be achieved
safely?
|
|
|
Animal
studies?
Is the claim for the promise of a drug or supplement based solely
on animal studies? While animals are useful for preclinical testing of new drugs,
there are many differences between animals and humans; and drugs
that show promise in animals do not always work, or work safely, in
humans. |
Summary: Weighing Sources of Medical Evidence
Weight
|
Description with notes
|
Strongest
|
Phase
III randomized blinded multi-center studies
Strongest if studies had large numbers of participants, and the
results are confirmed by other studies by independent
groups. Study outcome data is subjected to impartial independent 3rd-party
review, such as FDA.
Systematic Review is a study of studies, including randomized types. A systematic review may provide strongest evidence to guide clinical practice.
Strongest if studies had large numbers of participants, and the results are confirmed by other studies by independent groups. Look for the reputation of the journal - best if published in highly respected, peer-review journal. Best results will show low p-values and narrow confidence intervals. P-values < .05, or lower is considered statistically significant. Randomized studies do not show best use, necessarily, but provides best information about risks and benefits for a specific condition and setting. |
| Moderate |
phase
II studies - Small studies that look for
indications of clinical benefit. Determine if phase III testing is worth effort and expense, and importantly, if potential benefits offset risks to future study participants. Phase II studies are often not conclusive. Watch for sponsor hype
in press releases.. If safety profile is good, data from phase II studies might provide rationale for participation in a clinical trial. |
| Modest |
case
studies - anecdotal reports made by physicians, usually for off-label
use of an approved drug, for which there is existing safety information.
Hypothesis forming. Rationale for use is often mechanism-based.
Note that the rationale for Hormone Replacement Therapy was
supported by observation and case reports, and proven incorrect
in a controlled study. |
| Low |
phase
I studies - first dose finding studies in patients.
Cautiously seeks bioavailablity and safety information with
close supervision of participants. Determines if drug goes
to target cells, or organs of concern.
Hypothesis forming. Most drugs fail at this phase, but
even the great ones start
here. Typically participants have tried and failed other
options. |
Lower
Still |
animal
studies
- provide (imperfect) models for what
happens in humans.
Ya gotta start someplace.
Provides clues about toxicity and bioavailablity. Tumors are often transplanted into
animals - setting up an artificial host/tumor environment.
Animal studies can't account for important differences in biology,
metabolism, tumor/host interactions in humans.
|
| Lowest |
cell
culture (in-vitro) - show affects of compounds on cells
obtained from cultured cell lines. Hypothesis
forming. Where research starts; not what conclusions
can be based on.
Lacks information on bioavailablity, toxicity, activity in the
body.
Cancer cells are changed by being removed from the host
environment.
Even malignant cells will die spontaneously in cell
cultures. Only one in 5,000 preclinical agents make it to
the patient. |
| Off
the Charts |
testimonial -
Anecdotal reports made by individuals about improvements in
health associated with an intervention. Testimonials are
typically used as a marketing strategy on commercial sites; they are often a red
flag that studies have not been done. Testimonials
do not inform on: (1) who is reporting the result; if they have biases or conflict of interest - or if it is truthful account; (2) the case details, such as prior or subsequent treatments, or how the reported benefits were measured, or how long
the reported benefits lasted. (3) The background, such as the natural history of the disease.
Testimonials are typically selected by a sponsor … There is no way to know if negative reports are
excluded. |
| NOTES: |
Lack
of evidence does not mean lack of value. Even the most potent therapies
for lymphoma were in phase I level of testing at one time. |
References and Related Articles
-
Background articles on Evaluating Medical
Information - PAL
-
The limits of evidence-based medicine. Respir
Care. 2001 Dec;46(12):1435-40; discussion 1440-1.
Review.
PMID:
11728302
-
The Product Pipeline and Clinical Trials: Bringing a Drug to Market
- biology.iupui.edu
September 6 and 8, 2005
|
