Deciding what to believe: a scientists view
The confusing world of lymphoma with its myriad of treatment choices places us in the difficult position of deciding what we believe will be best for ourselves. There are many ways to do this. When faced with statistically equally-effective treatment choices putting ones faith in a trusted doctor, prayer, or a cutting edge research idea may all be valid ways to come to a decision. One thing I have learned over the past two months, since my oncologist encouraged me to find my way to a positive attitude, is that these decisions must come from a method that we find personally satisfying. Otherwise, we will never achieve the peace with our choices that can allow us to find that positive attitude.
I am unable to tell anyone how to pray or what doctor to trust, but as I have been traversing the myriad of advice available on the internet and in the medical journals it has reminded me of some of the basic principles of judging scientific research. I hope sharing these principles may help others when reading scientific results in their search for their own right choice. Simply put, the rest of this post is about how scientists decide what to believe. As background for my qualifications to pontificate on this subject I've spent the last 25 years involved in all stages of the scientific publishing process: author, reviewer, and journal editor. These experiences have taught me one important lesson: the scientific literature is not a repository of truth. Instead, the journals and abstracts are an ongoing conversation that attempts to further the progress of our fields toward a better understanding of the world around us. But that conversation contains plenty of ideas that turn out to be wrong. So, as we rely on the scientific literature to chose treatment plans we need the skills to separate the good from the bad.
There are three elements that scientists use to decide what to believe. These elements are understanding the mechanism that controls a process, statistical tests, and peer review.
Understanding the mechanism that controls a process seems to trump everything else. When someone believes they understand the process they are often willing to use that understanding to infer other conclusions. That is one reason why researchers struggle to understand the underlying mechanisms even when statistical evidence seems to prove that something works. For instance, no one fully understands how Rituximab kills lymphoma cells in all cases. We know the antibody attaches to an antigen on the cells and there is strong statistical evidence that it often kills the cells and there are explanations for why this works in some of the cases. But, no one really knows how the process completely works and so there is still active research on the topic and if someone figures it all out they will probably be able to find ways to make it work better. So, understanding the mechanisms is very important but not required. If the mechanism is understood a paper may not include the statistical tests discussed below but will skip directly to the peer review filter discussed even further down.
I say understanding the mechanism isn't required because everyone agrees that based on the statistical evidence Rituximab is a wonderful therapy and many of us choose to use it. So, how does statistical evidence work? Most scientists work with a standard known as 95% confidence. Just as it sounds this indicates that we accept something as true if we have 95% confidence that it is true. Turned around we accept a 5% risk that things we believe are actually false. This 5% is what we often see in abstracts or papers as the p-value. As in p=0.05 would be a 5% risk, p=0.01 would be a 1% risk. So, low p-values are a good thing. But, there need to be a couple of cautions when it comes to interpreting p-values as evidence of truth. The calculation of p-values requires comparing two ideas against each other. One is the idea the authors want to test and the other is generally a well-accepted idea that they want to improve on. In clinical research this well-accepted idea is generally that the placebo or control therapy is as good as the therapy being tested. So, the p-value depends on the chosen control. Pick a really stupid control and the p-values will drop unreasonably low. The tests also require a statistical model that does the comparison between the new idea and the control. If you pick a bad statistical model, the p-values will often be extremely low despite being meaningless. To be honest, I think clinical research is pretty good when it comes to selecting controls and statistical models. In prospective studies, the controls are randomly chosen groups of patients and that is considered the best way to figure things out. In retrospective studies, they have to accept comparing groups of patients that got different therapies for reasons that weren't random and so retrospective studies are generally given less credence. On the good side, retrospective studies let you test ideas without waiting years and years for new randomly chosen patients to go through treatment and future developments. In both types of studies, the statistical models being used are sufficiently standard and well understood that the p-values are pretty believable.
But, there is a bigger problem with believing things just because low p-values imply a good statistical result. You also have to consider a problem I will call "winning the lottery." Now, generally winning the lottery is not a problem. In fact, I'd be happy to win the lottery. The problem happens when I win the lottery and then you believe I know the secret behind how to do it and follow my advice. Scientific research sometimes is like the lottery with lots of players. At scientific meetings like the ones we hear about when studying up on therapies, literally thousands of abstracts are submitted and are accepted for presentation and publication with almost no review process. If you are taking a 5% risk of believing something that is false, then for every 1000 abstracts that depend on statistical tests there will be 50 that are actually wrong but appear to be correct. The problem is probably even worse because people tend to only submit abstracts with interesting results. So, lots of ideas that failed the tests are never submitted and lots of hypotheses are discarded before even being tested. So every 1000 abstracts really represents the best of an even larger pool of ideas. So, probably even more than 50 of the apparently good ideas are wrong. That's a pretty scary idea when you are looking over the recent abstracts and trying to pick a treatment path. But, it is true and you should be aware of it. Sometimes believing a single abstract may be a good idea. Maybe it looks like a great study because it has a large pool of patients (large numbers are really helpful). Maybe it looks like a great idea because there is some understanding of the mechanism. Maybe it looks like an idea worth trying because there is little risk. Those are all possible reasons to choose to believe that an abstract's results are worth following. But, if there is time between when an idea is first discovered and when you need to judge it then the process moves on and the pe!
er review process can help us out.
If a study is truly good, then the submittal of an abstract to a meeting should either be preceded or quickly followed by submitting a paper to a journal. Here the peer review process helps filter out the good ideas from the poorly done studies. An editor at the journal first decides if the topic is appropriate for the journal and then tries to get three other researchers to review the paper. In many fields only two reviewers are used, but the medical literature is quite careful and generally uses three reviewers. The goal is to find people who have the knowledge to comment on the paper and who represent a range of opinions on the topic. If the paper addresses a well-known controversy you will try to find reviewers from both sides of the controversy. Otherwise you look for people who have done similar work. The reviewers return detailed comments to the editor who then decides to either accept the paper as is (rare), reject the paper (should be less than half the time but is pretty common), or request revisions to address the comments of the reviewers (the most frequent outcome). These revisions can either be minor to address the clarity of the paper or major when they require significant amounts of additional research. In the case of major revisions the paper may be sent back to the reviewers for additional comments. Generally, once revisions are requested for a paper it will eventually get accepted for publication. You try not to put people through the revision process unless you expect that the paper can be made acceptable for publication.
Even once a paper gets published, people may find flaws in it. Often these are discussed in future papers, but sometimes they are sufficiently important that someone will write a comment on a paper. The authors then write a reply and the comment and reply go through the review process and may eventually wind up in the journal. So, the journal really becomes an ongoing slow, but hopefully thoughtful, conversation.
This peer review process is one of the reasons scientists tend to be very skeptical and spend a lot of time questioning ideas. It can be very hard to have every idea you put forward subject to intense scrutiny, but we are trained to do this because we really believe it helps separate the good ideas from the weak ones. But, it really isn't personal. Ok, sometimes it does get personal. For instance, I once had a reviewer call my writing style "a cruel form of mental torture." Then it's the editor's job to detect this and get the process back on track.
So, this process is much slower than presenting an abstract at a meeting but most abstracts should result in a published paper in the next year or two. If not, something could be wrong. It could just be that the authors are really busy with other things (I've been guilty of that). But, it could be that the paper was rejected due to reviews that uncovered serious flaws in the logic. And here's the kicker, even if the paper is rejected and the authors agree it was a bad idea the abstract will remain in print and in the online databases. So, if you see an idea cited to an abstract that is more than a few years old and there has been no follow-up paper it is time to worry about the idea and do more work to find out what happened.
Another good sign is if there are several abstracts on a topic. Often someone will make a convincing presentation at a meeting and others will go back to their labs and start working on similar ideas. An idea that was convincing at one meeting may be followed by several similar abstracts in the next year. That is a good sign that people were convinced and the follow-on studies reduce the likelihood that an incorrect idea has accidentally passed the statistical tests.
There is also an informal review process at meetings based on conversations in the hallways and during coffee and beer hours. If someone presents something exciting, it will be discussed. If someone has a good reason to discredit the idea, that tends to get around too. So, when one of our doctors comes home from a meeting excited about a new idea it may be based on more than just reading the abstract and hearing a short presentation.
Yet, another good sign is that if the idea is judged sufficiently important and many papers are starting to be written on the topic, eventually someone will be asked to write a review paper that summarizes all of the important research on the topic (both positive and negative) and tries to put the results in proper perspective of other research in the field. These review articles are also subject to peer review and so can be a really good place to find out how a field is progressing.
So, as a scientist turned patient dependent on a field I really don't understand, what do I chose to believe and what do I treat with skepticism? Review papers in major journals come at the top of the list, but by definition these are just slightly behind the cutting edge. Ideas where the mechanisms are understood are great. Published studies in peer-reviewed journals are good. Ideas supported by multiple peer reviewed papers are great. If you replace "publications in peer-reviewed journals" with "abstracts presented at meetings", my belief goes down a notch. At the bottom come ideas supported by a single abstract that had a small pool of patients and that wasn't followed by publication in a journal and that didn't inspire other studies. When you see something like that being used to support an idea it is time to look further before believing. Maybe there are better references to support the idea and you just haven't been told what they are. But, maybe that isn't the case and the ideas in the abstract were later disproved.
This is a tough topic. We are faced with the prospect of applying rapidly changing scientific progress to the goal of extending our lives as long as possible with as great quality of life as possible. Sometimes it will be worth taking a shot on a really new idea. So, please don't think I am trying to talk anyone into being very conservative and only accepting standard therapies that have been discussed in review articles. I just hope that knowing a bit more about the real process of science will help you decide when to be conservative and skeptical and when to take a shot.
And finally, remember that all of our reading and research is a way of having more productive conversations with our oncologists. So, we can always get them to help us through these scientific judgments too.
~ Andy (advisor to PAL)
