Close up of old English dictionary page with word evidence

By Ari Grubner, MD

Type “ketchup” into the Amazon search box and you will get 510 results. 510 for a tomato paste. If you’ve attempted to purchase anything online recently, you’ve likely experienced over-choice — a phenomenon in which people have a difficult time making a decision because they’re faced with too many options. Do I want sugar-free ketchup? Perhaps organic? Brand name? Generic? The upside-down plastic bottle is certainly practical, but aren’t the old-time glass bottles more classy? When experiencing over-choice, making a decision suddenly becomes overwhelming due to the many potential outcomes and risks that may result from making the wrong choice.

Now, the particular brand of ketchup that adorns your dinner table may not be all that important. But there are numerous choices that involve the big life decisions as well. When buying a house, nobody’s criterion is: “It should be a house.” There are different types of houses with different qualities, and of course, at different prices.

In the world of medicine, our “ketchup” is evidence. If you want to prove a point, promote a treatment, or convince your colleagues that a certain medication is more effective than another, you’ll have to prove it. And, similar to ketchup, physicians recognize that evidence comes in all shapes and sizes. This may appear to be a self-evident fact (the idea that there are different levels of evidence), but it seems that many people are unaware of this.

I was recently discussing the COVID-19 pandemic with a friend and he pointed out that “masks trap bacteria in our mouths and cause infections.” I found this to be an interesting point and asked, “Wow. Where did you hear this from?”

“Oh, I just came up with it, and it made sense to me.” Non-expert opinion. The lowest form of evidence (i.e., no evidence at all).

Another example: my neighbor told me about a local woman who professes to be able to heal strep throat with unpasteurized goat’s milk, gargled in the throat. He swears that he tried it himself and it seemed to work. Now this level of “evidence” is still quite weak, but it’s actually one step better than the friend who feared that his mask was causing bacteria. The anti-masker had no evidence at all — it was merely a person (a non-expert, in this particular case) stating his opinion. He had no proof to it. In this second example, the person is claiming that it worked. It’s not merely an opinion but a fact (assuming he is trustworthy). In medicine, we call this “anecdotal evidence” — evidence collected in a casual manner, which relies entirely on personal testimony. It is not, by any means, a well-designed study, but rather a single person’s experience, where something seemed to work. Weak evidence, but evidence nonetheless.

The issue is: How does this person know that the goat’s milk truly helped heal him? Maybe he was doing something else that made him better. Or maybe he would have gotten better whether he took the goat’s milk or not. His “study” lacks a control group.

The highest, most powerful form of evidence in medicine is a study performed with a control group. This means that in addition to testing the medication on a group of sick people, a second, similar group of sick people doesn’t get the medication and those individuals are studied as well. This allows the experimenter to see if it’s the drug/intervention that is truly making the participants better (i.e., if everyone who received the medicine gets better, and the control group does not get better).

Another, quite obvious factor that doctors look for when evidence is presented is the number of participants in a study. Some studies, for example, can have hundreds of thousands of people enrolled! When presented with such a study vs. the goat-milk healer, who would you go with?

In the Gemara shiur that I attend at night, the maggid shiur noted that attendance had been slowly declining over time. One night, after the shiur, he handed everyone a piece of paper to take home. I was stunned by what I saw. It was a survey with 10 questions to fill out. Some of the questions included:

  • What time is realistic for you to come to the shiur?
  • What are your suggestions for improving the shiur?
  • Do you have a chavrusa to review what is learned in the shiur?

This was a maggid shiur performing a scientific study! He was trying to collect data and make evidence-based changes to help improve the shiur. He could have simply gone over to one person after davening and had a discussion about the waning attendance of the shiur. But he recognized that data does not equal data. One person telling his opinion is different than collected data from the entire group of people involved. He decided to conduct a study, get evidence, and make changes based on that.

There is much more to be said on this topic, but suffice it to say that evidence does not equal evidence. It is fascinating to me that many intelligent, thought-out people seem to rank all forms of evidence as equal. They will trust their neighbor’s advice (non-expert opinion) over a large clinical trial with a control group.

There are certainly times at which high-level evidence is not available. Imagine, for example, the heart-breaking case of a patient of mine who had a rare genetic illness of which there are only 10 reported cases in the world. When dealing with such a condition, there certainly won’t be high-quality studies with a large number of participants and a control group. At best there will be very small studies, or perhaps even a lower level of evidence — “expert opinion” — a recommendation (without any hard data) formulated by a group of people with clinical expertise, which is based on everything that they know about the subject.

I recognize that we can’t use evidence for every life decision. When I want to know the best route to JFK airport, I will not hesitate to ask the guy sitting next to me in shul for his opinion (non-expert opinion). We all make decisions like that. Sometimes we don’t have the time to research the topic, or sometimes that is the only available “evidence” that we have. But when it comes to big life decisions shouldn’t we give weight to how strong that evidence is?

I am not suggesting that we be rigid and calculated in every movement we make. We are certainly allowed (and encouraged) to get layman’s advice from others to help guide life decisions. But let us recognize such advice for what it is — advice. We live in a time when we have access to massive amounts of data. Don’t make big life decisions based on someone telling you his opinion in a convincing or impassioned way. Do the research. Collect the data.

And please, please, don’t use goat’s milk to heal strep throat.

Dr. Ari Grubner studied in Ner Israel Rabbinical College and then completed medical school at the University of Maryland School of Medicine. Dr. Grubner is the host of a podcast that highlights and shares the wonders of G-d’s creations and the human body, “The Wonder — Niflaos Haborei Podcast” (anchor.fm/arigrubner). Dr. Grubner is currently a resident in pediatrics and lives in New York with his wonderful wife and their three sons. He can be reached at NiflaosHaBoreiPodcast@gmail.com.

2 COMMENTS

  1. How does one guage evidence provided by scientific studies when there are many studies which are clearly influenced by political agenda (see “climate change”)?

  2. What do you do when very many of the same “experts” also claim that according to “science” there are more than two genders?

LEAVE A REPLY

Please enter your comment!
Please enter your name here