Be careful interpreting scientific studies

[Gerry Seymour]

There's a problem I keep finding in how people (including the media - perhaps especially the media) react to scientific studies. They jump to conclusions, saying coffee protects against Alzheimer's, because a study (one study) finds a negative correlation between coffee consumption and Alzheimer's. They draw conclusions that are far beyond the scope of the study (a recent NPR report talked about a study showing that a person's natural running gait is their most efficient - which is neither what the study concluded, nor what it tested).

So, a few thoughts for consuming scientific information:

1. Consider the source. There's a fair amount of pseudo-scientific information out there, and many websites (especially, though mass media sometimes falls into this trap) don't differentiate between an uncontrolled and unreviewed study, and one that was performed with strict control groups and was reviewed and found reliable by experts in the field.
2. Don't get too excited about one data point. One study can be very incorrect in its conclusions. At the end of every study, the researchers draw some conclusions from what they discovered. But a single study will always be flawed - it's simply impossible to account for all necessary controls in a single trial. So, wait for more studies, and preferably the meta-analysis of several studies.
3. Pay attention to what they actually studied. Let's go back to that running gait study I mentioned earlier. Researchers tested runners - both experienced and entirely untrained (non-"runners"). They had them run using their usual (note: not necessarily "natural", which was the term NPR used) gait, versus some other gait. They found O2 efficiency highest with the gait they were used to. What they didn't study was whether the participants could become more efficient by learning a new gait and practicing it for an extended period (that would be a new study), but NPR's report concluded that our natural gait is our best gait.
4. Pay attention to the population used. Some studies exclude women, so they don't have to try to correct for monthly hormonal changes (which would require a longer study with more participants, which is much more expensive). Some only study women. Some only include pregnant women. Some use mice.
5. Look at the level of exposure used. There are studies that show direct, irrefutable carcinogenic effects of some chemicals. Mind you, they fed mice the human-equivalent of a pound or so of the chemical a day to get that outcome, so it's possible any reasonable dosage would be harmless. We know too much sun can lead to skin cancer, but nobody should believe you're going to catch cancer walking from your house to your car.

There are other things to watch for, but the main message is this: don't get too reactionary. Science is rarely speedy, because it takes time and effort to get enough well-controlled results to draw rational conclusions.

Any other thoughts or suggestions on how to intelligently consume scientific results?

 

[jobo]

there a book in my collection called bad science, which covers this topic in great detail, i can recommend it as,a good and very informative read
Bad Science (book) - Wikipedia

 

[jobo]

on your wider point, i just ignore all health scares and new research at least as published in the press, if i really want to know about it,i track down at least a summary of the study.

other than that i consume as much caffeine, fat, sugar, white bread, red meat, nicotine, milk etal as my body tells me it it wants, though i have tried to stop eating a litre of choc chip ice cream every night, with out to much success. My body wants it, it gets it. I gave up alcohol when i was very skint, and no matter how i try to start again, it just makes me ill.

I'm in the sobering position of knowing i will die some time in the next 25 years, so I'm enjoying it while i can.

that said, I've never been healther than i am now, if you measure health by feeling good and not as fit as now since my late 20s,
yes health scares can go take a jump, kill joys

 

[JowGaWolf]

There's a problem I keep finding in how people (including the media - perhaps especially the media) react to scientific studies. They jump to conclusions, saying coffee protects against Alzheimer's, because a study (one study) finds a negative correlation between coffee consumption and Alzheimer's. They draw conclusions that are far beyond the scope of the study (a recent NPR report talked about a study showing that a person's natural running gait is their most efficient - which is neither what the study concluded, nor what it tested).

So, a few thoughts for consuming scientific information:

1. Consider the source. There's a fair amount of pseudo-scientific information out there, and many websites (especially, though mass media sometimes falls into this trap) don't differentiate between an uncontrolled and unreviewed study, and one that was performed with strict control groups and was reviewed and found reliable by experts in the field.
2. Don't get too excited about one data point. One study can be very incorrect in its conclusions. At the end of every study, the researchers draw some conclusions from what they discovered. But a single study will always be flawed - it's simply impossible to account for all necessary controls in a single trial. So, wait for more studies, and preferably the meta-analysis of several studies.
3. Pay attention to what they actually studied. Let's go back to that running gait study I mentioned earlier. Researchers tested runners - both experienced and entirely untrained (non-"runners"). They had them run using their usual (note: not necessarily "natural", which was the term NPR used) gait, versus some other gait. They found O2 efficiency highest with the gait they were used to. What they didn't study was whether the participants could become more efficient by learning a new gait and practicing it for an extended period (that would be a new study), but NPR's report concluded that our natural gait is our best gait.
4. Pay attention to the population used. Some studies exclude women, so they don't have to try to correct for monthly hormonal changes (which would require a longer study with more participants, which is much more expensive). Some only study women. Some only include pregnant women. Some use mice.
5. Look at the level of exposure used. There are studies that show direct, irrefutable carcinogenic effects of some chemicals. Mind you, they fed mice the human-equivalent of a pound or so of the chemical a day to get that outcome, so it's possible any reasonable dosage would be harmless. We know too much sun can lead to skin cancer, but nobody should believe you're going to catch cancer walking from your house to your car.

There are other things to watch for, but the main message is this: don't get too reactionary. Science is rarely speedy, because it takes time and effort to get enough well-controlled results to draw rational conclusions.

Any other thoughts or suggestions on how to intelligently consume scientific results?

The media and many of it's consumers don't like detail so much of what was in a study is left out and assumptions are made for "easy digestion." There have been times when I've seen or heard reports where the reporter summed up the story and got it all wrong. The first thing that came to my mind was "That's not what the guy just said." This often happens when the reporter has little or no knowledge on the subject that they are reporting about. In my opinion I think the best reporters to report on studies are those who have experience or who have studied in the same field that they are reporting on.

 

[jobo]

The media and many of it's consumers don't like detail so much of what was in a study is left out and assumptions are made for "easy digestion." There have been times when I've seen or heard reports where the reporter summed up the story and got it all wrong. The first thing that came to my mind was "That's not what the guy just said." This often happens when the reporter has little or no knowledge on the subject that they are reporting about. In my opinion I think the best reporters to report on studies are those who have experience or who have studied in the same field that they are reporting on.

the media want a snappy head line and a very simplistic mesage, they also like to report bad news as it gets more attention than good news, between the two health scares are good copy

 

[Buka]

Unless I see it on the cover of one of those newspapers they sell in the checkout lines at stores.....

 

[Gerry Seymour]

Unless I see it on the cover of one of those newspapers they sell in the checkout lines at stores.....

Well, obviously! Those are the only really reliable sources.

 

[hoshin1600]

Another thing that should be obvious is that the media has a bias. I don't trust the media to tell me the weather outside much less anything else.
Next the studies themselves can be biased. If they received money from the tabbaco lobby to prove smoking is good for you and the results show different, guess what...your funding is going to disappear.
We all like to think science is facts but the truth is that we are in an unprecedented period of bogus science.

 

[JowGaWolf]

Another thing that should be obvious is that the media has a bias. I don't trust the media to tell me the weather outside much less anything else.
Next the studies themselves can be biased. If they received money from the tabbaco lobby to prove smoking is good for you and the results show different, guess what...your funding is going to disappear.
We all like to think science is facts but the truth is that we are in an unprecedented period of bogus science.

That is true. The bogus science is easier to access thanks to the Internet and YouTube. I always get a good kick out of the flat earth society videos that I come across. I'm definitely exposed to more crap since the birth of the Internet and social media.

 

[Gerry Seymour]

That is true. The bogus science is easier to access thanks to the Internet and YouTube. I always get a good kick out of the flat earth society videos that I come across. I'm definitely exposed to more crap since the birth of the Internet and social media.

That's one I just don't get. There's so much easy evidence to counter the idea that the Earth is flat...

 

[Danny T]

Question everything.

 

[JowGaWolf]

That's one I just don't get. There's so much easy evidence to counter the idea that the Earth is flat...

They are a special breed of humans. Their is a global movement where the older the information is the more validity it has a the truth. This logic bleeds into everything from science to health and religion. Just expect more of this messed up logic in the future. It hasn't peaked yet.

 

[JowGaWolf]

Question everything.

Questioning is fine. I question people's logic all the time. lol

 

[Gerry Seymour]

They are a special breed of humans. Their is a global movement where the older the information is the more validity it has a the truth. This logic bleeds into everything from science to health and religion. Just expect more of this messed up logic in the future. It hasn't peaked yet.

That particular logic shows up in MA, too.

 

[Monkey Turned Wolf]

They are a special breed of humans. Their is a global movement where the older the information is the more validity it has a the truth. This logic bleeds into everything from science to health and religion. Just expect more of this messed up logic in the future. It hasn't peaked yet.

The opposite is also true sometimes though. People will discount anything old by saying 'its not modern', even if there's validity to the concept.

Looking at when an idea came about to determine validity is never a good idea. Much better to look at the idea itself.

 

[jobo]

That's one I just don't get. There's so much easy evidence to counter the idea that the Earth is flat...

there is no evidence that you can see with your own eye, that its a ball, unless you go up to orbit, as with so very much of science it is dependent on you believing what you are told by another and some times quite often in fact, it turns out the cleverest men in the world were wrong, some times a bit wrong, some times very wrong

did you know, for instance, that our earth it at the exact centre of the observable universe, right in the middle, just like it says in the bible