Evaluating Evidence: Oncologists Do It Best

NormalThere is one crucial area of thought, one discipline where the vast majority of people could really use some education, and they are not getting it. You can see it everywhere. Some of the world’s most prominent and respected people are just really bad at it, have never been taught how to do it, and for the most part, do not need it.

Or they don’t think they need it, but a lack of understanding in this crucial area is leading to all manner of problems.

Evaluating evidence.

It sounds easy enough. We do it every day. We take in data, and try to predict what is going to happen, we get it wrong and reassess, and then stumble along doing it all over again.

Gradually, some enterprising individuals learn to put some order into it, learning that cloudy days may require an umbrella, or a flirtatious look and smile may be the beginning of a beautiful relationship. We come by that sort of thing naturally, without thought, but the lack of thought is what gives us away.

There are days when the cloudy skies mean snow, and days when that ‘look’ was for the guy behind you.

Politicians and Intellectual Ability

Political animals usually are smarter than average. The defined mean of Intelligence Quotient is 100, and most people who get into high school (though not necessarily out) have at least this much. There is a broad distribution, estimated by the typical ‘Bell Curve’ or normal or Gaussian distribution depicted above. Such probability distributions describe the variations of intelligence (and many other randomly organized characteristics), although the skewing of the underlying shape can look quite different. Generally, there is a lot of the population around the mean (average), tapering out to much smaller numbers at either end.

People reading about this often become defensive, thinking somehow that they are responsible for whatever level of failure or success they have in this area. While I know the brain is like a muscle in many respects, the more you think the smarter you get, most people are completely innocent of their particular level of intelligence. Knowledge is another thing, but even with knowledge, the level of acquisition of knowledge is related in many ways to what you were born with, and there is not a lot you can do about what you were born with.

People with IQs of 80 or less should not feel guilty about their abilities, any more than short people should blame themselves for not being able to dunk a basketball. But since only ten to fifteen percent of individuals have IQs in this range, for most it is not really an excuse to misunderstand the evaluation of evidence if they work at it a bit.

Politicians have the ability to understand evidence, as does the vast majority of the population.

But it is what you do with what you’ve got that is important, not what you were born with.

Evaluating Evidence Can be Tricky

There are, of course, wide variations in success of evaluating evidence, which do trend toward a correlation with intelligence, but if you never focus on the issue, lots of times you don’t know what you don’t know. Necessity, though, is the mother of invention.

Years ago, physicians caring for people with malignancy had a hard time figuring out what therapies helped, and what didn’t. In some disciplines, success of treatment can be recognized by simple observation. Common practice demonstrated pretty quickly that immobilizing a fracture was at least helpful for the pain, and putting bones back into alignment, and then keeping them there, was key to a functional recovery.

Obvious. Double blind clinical trials were really not terribly necessary for such a gross end result. To some extent, this type of observational study is anecdotal, but since it happens many times the world over, and the anecdotes add up, the evidence starts to approach a quality that is reliable. There are fundamental mathematical explanations of this, but it is also intuitively obvious that larger numbers of observations almost always lead to greater confidence in the conclusion. Almost.

And that’s one place where is gets tricky. But I’ll come back to that.

The Trouble With Oncology

Cancer is one of that last areas of medicine to bow to therapeutics. There is always a level of ‘noise’ when examining the results of any observation. Things happen to people in many random, or seemingly random, ways. It is not really random, but because the causes of variation are so numerous, it sure appears random.

People with cancer have life spans which vary considerably, even amongst those with the same disease, even amongst those diseases with the same stage. I routinely cared for patients with stage IV lung cancer, and when asked for timelines (which patients often request) average survival without treatment was four months, and with treatment they might get out to eight or ten months (it is finally getter better than that, by the way). We could tell them that on average, 90% would die within the year, 65% if they took treatments of some kind.

Notice that in medicine, physicians, especially specialists who can focus in smaller areas of interest and concern, give such answers in terms of probabilities. It has to be given with care and sensitivity, but, in my opinion at least, it ALWAYS has to be accurate. It is a rule of mine. You want your patients to believe you far more than you want them to like you. Never lie to the patient. Express doubt, even ignorance, but never lie.

I have had some patients (not generally lung cancer) live for ten years with metastatic disease, and others live only a matter of days or weeks. If you plot these kinds of events, such as survival over time, within this wide scatter is a trend. The more events you have, the clearer is that trend. At five years, virtually all lung cancer patients with stage IV disease on presentation have passed away. But around that trending curve of deterioration over that five years is incredible variation.

So how do you know, when a patient lives for two years on a specific treatment, that the treatment had any effect? If you compare two patients, one with the treatment and one without, it could end up either way. The problem was, with some treatments, the variation was so large (the noise), that even a hundred patients, or a thousand patients, was not enough to separate random fluctuation from actual treatment effect.

And that is about the same time that very clever people started to use sophisticated mathematics, probability analysis, and experimental designs, to establish a difference we could rely upon. Treatments in cancer were so poor, we could not see the differences within the noise, without very detailed and complicated analysis. To this day, at the extreme of complexity, understanding the science behind all of this is out of reach for any but the few who study such techniques all their lives.

The wide variations and poor efficacy of cancer therapies meant that oncologists had to learn a lot of these sophisticated techniques, and from my experience over the last forty years, we did so before lots of other specialties, because we had to. Otherwise we could not see the improvement over the background noise. Establishing clear proof was also necessitated by the fact that oncology treatments were so toxic. The physician’s maxim, “Primum non nocere,” or “First, do no harm,” was pretty difficult to follow in oncology.

As good as physicians doing this type of research get, it’s not good enough. Every major study using statistical techniques in medicine has a statistician or two in the background working with the authors to plan the experiment and evaluate the data. I entered medicine after an undergraduate degree in mathematics, with a lot of statistics in my repertoire, but I would never dare publish an article without formal professional statistical analysis by a statistician.

Pitfalls of Experimental Design

There are lots of ways, we discovered, of messing up an experiment or an observation. Just understanding the definition of randomness was a start. I remember well sitting in a seminar at the Princess Margaret Hospital in Toronto, forty years ago, where members of the team treating cancer patients were agonizing over the fact that survival curves of treated and untreated patients eventually fell back together…no matter how dramatic the initial improvement…until one observer pointed out that everybody dies eventually, so all survival curves come together. The survival curves were getting out to the normal lifespan, and nobody noticed that the results were competing with human age limits.

Studies,over twenty yeas ago, at my current centre showed that patients who underwent chemotherapy after surgery for esophageal cancer lived longer than those who didn’t. One might think this means chemotherapy helps. It actually does, but this study failed to show that because the two groups were selected retrospectively (not randomly or prospectively with a plan), and the groups were not identical apart from the treatments received. They were different in age by an average of ten years, for one thing, and the group that started ten years older did not live as long, of course. Additionally, it was recognized that patients who failed to adequately recover from the surgery they went through did not then always receive chemotherapy (they were too sick), thus biasing the results towards chemotherapy (basically proving only that sick people don’t do as well as healthy ones).

This last point is subtle. It basically noted that patients who got chemotherapy did better, but also noted that patients who got chemotherapy started off better. Such problems in experimental design have lead to concepts demanding similar populations and analysis based on the original ‘intent to treat’ (patients who are planned to get a treatment get counted in the treatment group even if life’s problems prevent them from starting or completing it).

The analysis of this study was clearly flawed by the selection bias and the age difference…something that the vast majority of people might easily overlook, but that Medical Oncologists are geared to, and trained to, understand. It was also NOT a randomized prospective study, it was an observational study looking back at a bunch of patients. When you look closely at the data, it basically proved that patients who do well with a complex treatment tend to do well. Not really very helpful.

Most experimental designs of clinical trials demand that the two populations being compared are ‘identical’ in every aspect EXCEPT the treatment they receive. If the control group is ten years older than the experimental group (as occurred in that study of patients with esophageal cancer above) one could expect they would die ten years sooner, thus confounding the treatment effect. This is a clear example of bias that can creep in when you are not looking, and extremely intelligent people fall victim to this obscuring effect if they are not extremely vigilant. It becomes pretty clear that people who do not do this stuff all the time may fall prey to confounders and experimental bias. It is subtle. It is powerful.

After all, we are all human, and we really want to see good results.

Because lives depend on understanding this stuff, Medical Oncologists are constantly immersed in this type of evaluation of evidence. But you can understand that not everybody else is. Most treatment effects are less subtle, and most other professions and disciplines have far less profound effects when they make a mistake.

Second Nature

So it all becomes second nature to us, those who practice medical oncology. When someone argues that a comparison of two groups, or a correlation of two populations with subsequent events, suggests a cause and a direct resulting effect, we stand back and look for all the confounding biases. And to us, because we live with this stuff every day, its all pretty easy to see coming. Correlation does NOT equal causation.

A few weeks ago, during a debate between Bernie Sanders and Ted Cruz on health care funding, Senator Cruz argued that patients who obtained private health insurance lived longer than patients who relied on Medicaid. Bernie had no answer to this, it seemed, though in this context he may never have got a chance to respond, or he may have felt the explanation would be lost on the audience. To those of us who understand the relationship between health outcomes and socioeconomic status (physicians, nurses, social workers, health care administrators) the answer was pretty obvious, and not at all clearly related to the quality of insurance or the delivery of health care. People in higher socioeconomic class ALWAYS live longer: more money means better education and better health life style including food, drugs, alcohol and smoking issues. There is simply no need to introduce the insurance aspects to explain the difference, but that is exactly what the politicians did, because they don’t understand evaluation of evidence and experimental analysis.

If a physician, particularly a research physician, made this claim on this evidence, we might suspect fraud, so egregious is this mistake. We recognize he might be right in his conclusion, though totally wrong in how he got to it, and we might still call it fraud, or at the very least, lazy ignorance.

Trump Does Not Understand Evidence

One thing we teach our medical students is to protect their brains. If they read something somewhere, a year later they cannot possibly remember where that informational tidbit came from. So don’t read that stuff, we tell them. Sources of information vary hugely in their quality (so do sources of news).

We teach our students to evaluate everything carefully, but we understand that they must delegate this work to others from time to time. So it is that we teach them to rely only upon peer-reviewed journals of high impact and integrity. Journals in medical information gain a level of respect which allows us to accept their findings based on historical activity. Something published in the New England Journal of Medicine, for example, is well known to be highly credible because editors scrutinize the studies, and peers review the results and report in letters, meetings and other journals. The Journal (NEJM) struggles to maintain that reputation, and if they ever fail, they are outed VERY quickly. In fact, reputable journals like NEJM and The Lancet usually out themselves. The Lancet retracted the Wakefield article connecting vaccines to autism when it realized the level of fraud involved in its production. It already knew the work was inaccurate, but it was the deception that prompted the first retraction for that journal in its history. [see Vaccines… this blog] Inaccurate reports are worthy of discussion. Fraudulent reports are worthy of ridicule (and discussion…this one, among some others, lead to major changes in rules governing authors’ activity in publishing medical journal articles, and medical journal editors in accepting them).

So the reputable journals try very hard not to fail. They do that by having extensive peer review by experts in the field, by having internal evaluations, and perhaps most importantly by inviting and receiving ‘letters to the editor’ arguing for or against certain studies. And they publish opinion articles by experts which lay out arguments for and against articles that appear in their (and other) journals.

There are, unfortunately, many journals where that degree of professional integrity is not maintained. There are some, in fact, that have an preconceived agenda, either because they are actually produced by a pharmaceutical industry, or have a major connection to an organization with a vested interest in an ‘end result’. Some, for example, want to promote religious belief, and the power of prayer, or the power of Christian values. We know that, we know which journals carry this type of bias, and we take that into consideration whenever we evaluate evidence, starting with avoiding those sources completely. And we teach our students not to read those journals.

Because a year later they cannot possibly remember where that informational tidbit came from.

Many people do NOT understand this structure of evidence evaluation. Those of us who deal in evidence every day know that we have to start with an evaluation of the source of evidence, then of the logic of the evidence, then of the design of the experiment or observation. One thing upon which we all agree is that wonderful statement by Christopher Hitchens: “Anything which may be asserted without evidence, may be dismissed without evidence.” Dismissed as proof anyway. But idle chat and random thoughts, as well as disciplined thoughts, can and do lead to hypotheses to be investigated and tested.

It is somewhat similar to the another principle, this time in physics: “If the event cannot be observed in any way, it does not exist.” Physicists argue that if you cannot observe something, it is sort of pointless to talk about it.

Trump does not understand evidence. He is not alone, although many people seem to come by some form of evidence evaluation naturally, without formal education.

The source of information is clearly extremely important. Certain news agencies do not cut it because they have a history of not cutting it. “Future behavior is best predicted by past behavior.” If one of my very intelligent friends who never says anything without good supporting evidence, actually says something, I am more inclined to believe them than I am someone who never gets it right.

So when Trump uses politically motivated sources of information, it is suspect. When he uses sources that often get it wrong, it is even more suspect. When he uses Fox News, when they offer no evidence, and restates a meme without other independent evidence, his argument can easily be dismissed. Because in that context, if it is true it is completely coincidental.

The problem is that Trump never cites any evidence. He does not have the evidence to cite. Indeed, it seems he does not care about it. The people he speaks to are not waiting for evidence, they are waiting for him to make the same claim three times, usually in rapid succession.

“Fake news. Phony. Fake.” This was his claim at the CPAC speech. No evidence, just the claim. But he never points to an example. Indeed, often it is in response to corrections the media has made of his comments, pointing out his fake news, such as the claim of a landslide victory, which his election result clearly was not.

“Believe me.” He says this repeatedly, as an alternative to evidence. “Believe me.” Why? Nobody ever answers ‘why’.

“That Which May Be Asserted Without Evidence May Be Dismissed Without Evidence.”

So when Trump states that Hillary Clinton lied, if he cannot provide evidence, that statement should be dismissed. When he says some terrorist action occurred in Sweden last week, and provides no evidence (outside of saying “Someone said it,” or “I heard it from some people,” both very common Trump supporting statements), the thought he expresses should be dismissed. The big problem is that Trump NEVER accurately identifies his evidence, and you really have to assume he has none, if none is supplied. I have listened to him assiduously, to my extreme detriment, for the last year, and could count on my fingers the number of times he refers to some article, never to a source which is as reliable as a scientific or medical journal.

Trump makes such blanket statements all the time. “I inherited a mess,” he has said. No evidence. If you dig, the evidence is actually contrary. Trump says something is a disaster (well, he actually says this about everything) but never gives any evidence. Consider unemployment. Trump says unemployment is a disaster, though all apparent evidence suggests unemployment is at one of its lowest rates.

There are times he gets more sophisticated about his dishonest comments. The rate of change in murders jumped in 2015, in part because the rate was so low in 2014 (probably a reporting and clerical error, but murder rates are based on a huge number of variables). He jumped on that point to promote his fear riddled approach to his underlying desire to promote authoritarian rule through police and military. He overlooked the reality that murder rates (not rate of change) are massively lower now than they were thirty to forty years ago. In addition, the background ‘noise’, as in any ongoing observation of large numbers, does not allow us to figure out if this rate change is anything other than random variation.

But ‘cherry picking’ (taking results which are favorable to your agenda) is really the same as lying, if you know you’re doing it, or simply lazy, if you don’t.

Lies

If I tell you two plus two equals five, and I do not understand addition, that is not a lie. It is a mistake.

If I say two plus two equals five, and I know it equals four, that is a lie.

If I say two plus two equals four, and I really think, because I am not too bright, that it equals five, that too is a lie, even though the statement is correct.

If I say two plus two is five, and I know it is four but I say it is hyperbole or humor when someone catches me on it (but I say nothing if no one objects), that is a lie.

That’s what Trump does when he says Obamacare is a disaster. Or when he says Mexicans are rapists. Or all the other countless exaggerations for which he is forgiven in the right wing media that supports him. “Oh, that’s just Trump,” they say, dismissing complaints of lying because getting truth from your president isn’t as important as promoting the right wing agenda.

We humans project. We actually try not to lie, for the most part. We tell ourselves we never lie, though we know we do a little bit (“No dear, that hat looks lovely”). And we treat other people as if the potential for lying, real lying, is very rare. We are always surprised when it happens. The blatant lie right to your face, the ‘gaslighting,’…we simply do not expect or anticipate that, and much of our communication is colored by what we expect.

Evidence suggests that I have a pretty stable, confident brain that takes time to carefully decide things. And yet, thirty years ago I observed an event which I could not believe was possible, which I could not emotionally deal with. I pushed it out of my mind. I denied it. No one else was really hurt by this decision of my subconscious brain, and I hope that had there been any detrimental effects on others that my facing this event could avoid, I would not have denied what I saw. About a year later when the consequences of that event doubled down with other similar events, I remembered it, but it was the clearest, most powerful example of cognitive dissonance that I have ever seen, and it was all mine. To this day I find it hard to believe my brain is capable of that kind of self deception. It makes me shiver to think about it. The lesson is that we can deny what we do not wish to believe.

As a medical friend of mine suggests, “There are none so blind as those who refuse to see, none so deaf as those who refuse to hear.”

Accepting that someone can lie at every opportunity is too threatening to our world view to accept.

Professional Lying

Lying in medicine occurs very rarely. Even when is does, it may not be in the awareness of the authors. Generally, medical authors believe what they are concluding, even if they fudge the data a bit to support what they believe to be true. But when it does happen, as it did years ago in a specific breast cancer treatment, or in the retracted article about vaccines and a relationship to autism, it takes everybody by surprise, and it takes a long time to sort out. We project our honesty on to the person telling us stuff, and we simply do not expect anyone to lie all the time.

Lying in politics is probably far more common than in medicine (I’m not really denigrating politicians here; in medicine your thoughts are carefully written down and analyzed. In politics they can be ‘off the cuff, in the moment’ and getting it all as right as I describe here can be close to impossible), but mostly it amounts to making claims without supporting evidence, or ignoring data that doesn’t agree with your preconceived conclusions. Lying to shore up your own reputation, to support your exaggerated opinion of your own value, lying to make the claim you are better than everybody, at everything, is actually very rare. “I understand better than the generals do,” or “I understand…better than anybody.” When he says that, what are you thinking?

We humans are not used to dealing with someone who lies all the time. Even as I say this about Trump, who I know by now lies almost every day, I find it difficult to accept, to believe. We hear people right now simply saying that ‘that is Trump’ and setting the dishonesty aside. Rick Santorum, last night after Trump’s first speech to Congress, when cont=fronted with some of these lies replied that is was pretty minor stuff for a president.

When Trump stated there were crowds lining up to hear him at the CPAC (a ticketed event with no line-ups), he didn’t care that he didn’t know that. When he said he had the highest electoral college result in history, and was confronted with the truth, his response was simply to say he had heard this somewhere, the implication being that this was sufficient justification for this lie. “Some people say,” and “many people think,” are phrases he has used, phrases which are the closest thing to evidence he appears to cite.

I remember wondering what Trump would do when people started to see the patterns, started to recognize that nothing he says is reliable. I remember how long it took me to unblock that terrible event I observed.

But I should have known. Trump told us what he would do. He would counter-punch.

And so now Trump is attacking the news sources that are finally calling him out on all his lies. They are ‘fake news’ and he has banned the most reliable ones from a recent media event.

“Fake news. Phony. Fake.” He says it three times, often punctuated by, “Believe me.”

And that’s all the evidence he gives.

In truth, I do not know if Trump can evaluate evidence. The example of murder rates says he cannot. But the constant lying says evidence is not important to him. Maybe he can understand evidence. “I have a very good brain.” “Only I can fix it.”

Even a Medical Oncologist can see the problem with evidence here.

It’s not just that Trump doesn’t understand evidence. He just doesn’t use it.

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s