Be careful interpreting scientific studies

Discussion in 'The Study' started by gpseymour, Dec 19, 2017.

  1. gpseymour

    gpseymour MT Moderator Staff Member

    • Supporting Member
    Joined:
    Mar 27, 2012
    Messages:
    24,130
    Likes Received:
    7,109
    Trophy Points:
    448
    Location:
    Hendersonville, NC
    There's a problem I keep finding in how people (including the media - perhaps especially the media) react to scientific studies. They jump to conclusions, saying coffee protects against Alzheimer's, because a study (one study) finds a negative correlation between coffee consumption and Alzheimer's. They draw conclusions that are far beyond the scope of the study (a recent NPR report talked about a study showing that a person's natural running gait is their most efficient - which is neither what the study concluded, nor what it tested).

    So, a few thoughts for consuming scientific information:
    1. Consider the source. There's a fair amount of pseudo-scientific information out there, and many websites (especially, though mass media sometimes falls into this trap) don't differentiate between an uncontrolled and unreviewed study, and one that was performed with strict control groups and was reviewed and found reliable by experts in the field.
    2. Don't get too excited about one data point. One study can be very incorrect in its conclusions. At the end of every study, the researchers draw some conclusions from what they discovered. But a single study will always be flawed - it's simply impossible to account for all necessary controls in a single trial. So, wait for more studies, and preferably the meta-analysis of several studies.
    3. Pay attention to what they actually studied. Let's go back to that running gait study I mentioned earlier. Researchers tested runners - both experienced and entirely untrained (non-"runners"). They had them run using their usual (note: not necessarily "natural", which was the term NPR used) gait, versus some other gait. They found O2 efficiency highest with the gait they were used to. What they didn't study was whether the participants could become more efficient by learning a new gait and practicing it for an extended period (that would be a new study), but NPR's report concluded that our natural gait is our best gait.
    4. Pay attention to the population used. Some studies exclude women, so they don't have to try to correct for monthly hormonal changes (which would require a longer study with more participants, which is much more expensive). Some only study women. Some only include pregnant women. Some use mice.
    5. Look at the level of exposure used. There are studies that show direct, irrefutable carcinogenic effects of some chemicals. Mind you, they fed mice the human-equivalent of a pound or so of the chemical a day to get that outcome, so it's possible any reasonable dosage would be harmless. We know too much sun can lead to skin cancer, but nobody should believe you're going to catch cancer walking from your house to your car.
    There are other things to watch for, but the main message is this: don't get too reactionary. Science is rarely speedy, because it takes time and effort to get enough well-controlled results to draw rational conclusions.

    Any other thoughts or suggestions on how to intelligently consume scientific results?
     
    • Like Like x 2
  2. jobo

    jobo Grandmaster

    Joined:
    Apr 3, 2017
    Messages:
    7,818
    Likes Received:
    1,129
    Trophy Points:
    263
    Location:
    Manchester UK
    there a book in my collection called bad science, which covers this topic in great detail, i can recommend it as,a good and very informative read
    Bad Science (book) - Wikipedia
     
    • Useful Useful x 1
  3. jobo

    jobo Grandmaster

    Joined:
    Apr 3, 2017
    Messages:
    7,818
    Likes Received:
    1,129
    Trophy Points:
    263
    Location:
    Manchester UK
    on your wider point, i just ignore all health scares and new research at least as published in the press, if i really want to know about it,i track down at least a summary of the study.

    other than that i consume as much caffeine, fat, sugar, white bread, red meat, nicotine, milk etal as my body tells me it it wants, though i have tried to stop eating a litre of choc chip ice cream every night, with out to much success. My body wants it, it gets it. I gave up alcohol when i was very skint, and no matter how i try to start again, it just makes me ill.

    I'm in the sobering position of knowing i will die some time in the next 25 years, so I'm enjoying it while i can.

    that said, I've never been healther than i am now, if you measure health by feeling good and not as fit as now since my late 20s,
    yes health scares can go take a jump, kill joys
     
    Last edited: Dec 19, 2017
    • Like Like x 1
  4. JowGaWolf

    JowGaWolf Grandmaster

    Joined:
    Aug 3, 2015
    Messages:
    8,112
    Likes Received:
    2,454
    Trophy Points:
    263
    The media and many of it's consumers don't like detail so much of what was in a study is left out and assumptions are made for "easy digestion." There have been times when I've seen or heard reports where the reporter summed up the story and got it all wrong. The first thing that came to my mind was "That's not what the guy just said." This often happens when the reporter has little or no knowledge on the subject that they are reporting about. In my opinion I think the best reporters to report on studies are those who have experience or who have studied in the same field that they are reporting on.
     
    • Agree Agree x 1
  5. jobo

    jobo Grandmaster

    Joined:
    Apr 3, 2017
    Messages:
    7,818
    Likes Received:
    1,129
    Trophy Points:
    263
    Location:
    Manchester UK
    the media want a snappy head line and a very simplistic mesage, they also like to report bad news as it gets more attention than good news, between the two health scares are good copy
     
    • Agree Agree x 1
  6. Buka

    Buka Sr. Grandmaster

    • MartialTalk Mentor
    Joined:
    Jun 27, 2011
    Messages:
    10,546
    Likes Received:
    6,875
    Trophy Points:
    448
    Location:
    Maui
    Unless I see it on the cover of one of those newspapers they sell in the checkout lines at stores.....
     
    • Funny Funny x 1
  7. gpseymour

    gpseymour MT Moderator Staff Member

    • Supporting Member
    Joined:
    Mar 27, 2012
    Messages:
    24,130
    Likes Received:
    7,109
    Trophy Points:
    448
    Location:
    Hendersonville, NC
    Well, obviously! Those are the only really reliable sources.
     
  8. hoshin1600

    hoshin1600 Senior Master

    Joined:
    May 16, 2014
    Messages:
    2,858
    Likes Received:
    1,332
    Trophy Points:
    253
    Another thing that should be obvious is that the media has a bias. I don't trust the media to tell me the weather outside much less anything else.
    Next the studies themselves can be biased. If they received money from the tabbaco lobby to prove smoking is good for you and the results show different, guess what...your funding is going to disappear.
    We all like to think science is facts but the truth is that we are in an unprecedented period of bogus science.
     
  9. JowGaWolf

    JowGaWolf Grandmaster

    Joined:
    Aug 3, 2015
    Messages:
    8,112
    Likes Received:
    2,454
    Trophy Points:
    263
    That is true. The bogus science is easier to access thanks to the Internet and YouTube. I always get a good kick out of the flat earth society videos that I come across. I'm definitely exposed to more crap since the birth of the Internet and social media.
     
  10. gpseymour

    gpseymour MT Moderator Staff Member

    • Supporting Member
    Joined:
    Mar 27, 2012
    Messages:
    24,130
    Likes Received:
    7,109
    Trophy Points:
    448
    Location:
    Hendersonville, NC
    That's one I just don't get. There's so much easy evidence to counter the idea that the Earth is flat...
     
  11. Danny T

    Danny T Senior Master

    • Supporting Member
    Joined:
    Sep 5, 2002
    Messages:
    4,234
    Likes Received:
    2,259
    Trophy Points:
    388
    Location:
    New Iberia, Louisiana USA
    Question everything.
     
    • Agree Agree x 1
  12. JowGaWolf

    JowGaWolf Grandmaster

    Joined:
    Aug 3, 2015
    Messages:
    8,112
    Likes Received:
    2,454
    Trophy Points:
    263
    They are a special breed of humans. Their is a global movement where the older the information is the more validity it has a the truth. This logic bleeds into everything from science to health and religion. Just expect more of this messed up logic in the future. It hasn't peaked yet.
     
  13. JowGaWolf

    JowGaWolf Grandmaster

    Joined:
    Aug 3, 2015
    Messages:
    8,112
    Likes Received:
    2,454
    Trophy Points:
    263
    Questioning is fine. I question people's logic all the time. lol
     
  14. gpseymour

    gpseymour MT Moderator Staff Member

    • Supporting Member
    Joined:
    Mar 27, 2012
    Messages:
    24,130
    Likes Received:
    7,109
    Trophy Points:
    448
    Location:
    Hendersonville, NC
    That particular logic shows up in MA, too.
     
  15. Monkey Turned Wolf

    Monkey Turned Wolf MT Moderator Staff Member

    Joined:
    Jan 4, 2012
    Messages:
    7,251
    Likes Received:
    2,193
    Trophy Points:
    263
    Location:
    New York
    The opposite is also true sometimes though. People will discount anything old by saying 'its not modern', even if there's validity to the concept.

    Looking at when an idea came about to determine validity is never a good idea. Much better to look at the idea itself.
     
  16. jobo

    jobo Grandmaster

    Joined:
    Apr 3, 2017
    Messages:
    7,818
    Likes Received:
    1,129
    Trophy Points:
    263
    Location:
    Manchester UK
    there is no evidence that you can see with your own eye, that its a ball, unless you go up to orbit, as with so very much of science it is dependent on you believing what you are told by another and some times quite often in fact, it turns out the cleverest men in the world were wrong, some times a bit wrong, some times very wrong

    did you know, for instance, that our earth it at the exact centre of the observable universe, right in the middle, just like it says in the bible123
     
    Last edited: Dec 19, 2017
    • Funny Funny x 1

Share This Page