A walk through the ivory tower
There are two things that really disturb me in academia – poor correlation/causality conclusions and silly research. My morning news reading offered two good examples of each. The first comes from the Indianapolis Star which reports a New England Journal of Medicine study which found “Women who drank up to a glass of alcohol per day were ‘cognitively equivalent to being approximately a year and a half younger’ than teetotalers.” But all the study really found was a correlation between a drink a day and a better mental state. I assure you, though, that I could find an even stronger correlation between eating ice cream and drowning. Does that mean ice cream can inhibit swimming? Of course not, people simply swim and eat ice cream more in the summer. I don’t necessarily blame the Journal or the scientists for this misconception. I blame the sloppy reporter who drew a conclusion when only a correlation exists, and by extension the scientists who didn’t make the inconclusive nature of the research clearer to that reporter.
The second annoying item, silly research, comes from the Washington Post magazine. Prof Todd Zywicki bluntly called it “stupid academic research” and “one of the dumbest I have come across in some time.” Rather than try to summarize I’ll simply direct your attention to Prof. Zywicki’s sufficient takedown of the project. My point with these two items is to highlight how important it is to cast a discerning eye on everything you find in the news, even if it comes from professors working under the auspices of “academic research.”
Update: Reader “philospher” offers this critique regarding the cognitive benetifs of alcohol: “So, not only were the scientists quite clear about the difficulties of inferring causation, they also did an excellent job of doing exactly the sort of things you need to do in a retrospective design to enable you to, in fact, infer causation.”
10 Responses to “A walk through the ivory tower”
In particular, anything at all in so-called “education” research is more than suspect.
Hey, Eric, here’s a novel idea for you: before you go casting serious aspersions at other people’s research, why don’t you actually read the research yourself? Here are some quotes from the NEJM study:
http://content.nejm.org/cgi/content/full/352/3/245
In regression models, we considered the following potential confounding variables, possibly related to both cognitive function and alcohol intake: age at the time of the interview (continuous); highest educational degree (registered nurse or associate’s degree, bachelor’s degree, or graduate degree); a history of hypertension, high cholesterol levels, diabetes, or heart disease (yes vs. no); level of physical activity, measured in metabolic-equivalent hours per week (quintiles); age at menopause; use of postmenopausal hormone therapy (current, past, or never); use of vitamin E supplements (yes vs. no); body-mass index (the weight in kilograms divided by the square of the height in meters [less than 22.0, 22.0 to 24.9, 25.0 to 29.9, or 30.0 or more]); cigarette-smoking status (current, past, or never); aspirin use (once or twice per week, three or more times per week, or none); ibuprofen use (yes vs. no); scores for the mental health index (0 to 79 [low] vs. 80 to 100) and energy
(Eric: My sincere apologies — I had accidentally scrolled down earlier and thought I saw your name attached to this post. Everyone please consider the word “Joshua” find/replaced for “Eric” throughout the previous comment.)
Hey, Philospher, here’s a novel idea for you: why don’t you actually read my post carefully before criticizing it? I read the report in full. . . parts of it twice now, thanks to your comment. I say quite explicitly, “I don’t necessarily blame the Journal or the scientists for this misconception. I blame the sloppy reporter who drew a conclusion when only a correlation exists, and by extension the scientists who didn’t make the inconclusive nature of the research clearer.”
In other words, I know darn well what the scientists were saying, and they weren’t saying much other than a correlation exists. I’m blaming the jounalist for mischaracterizing the report. I’m also slightly blaming the researchers for letting the journalists walk away with that impression. But the real blame for that rests with the reporter. Read posts more carefully before you make a knee-jerk critique without actually reading what I wrote.
After reading his post, it seems to me that Prof. Zywicki failed fairly completely to understand the most basic aspects of the methods & aims of the IAT research. In particular, he makes a completely baseless (& scientifically clueless) speculation about the relationship between the main experiment (the time-response stuff) and the little questionnaire afterwards (about various organizational preferences). There’s nothing in the main literature on the implicit measures that makes the kind of stupid causal claims that Zywicki would be right in protesting against had, in fact, the researchers been making them. Which they aren’t — as would be clear to anyone who actually read either (i) the main articles in the literature or even (ii) the entirety of the WaPo article.
Since Zywicki’s post was profoundly ill-informed, perhaps I could invite Joshua (or any other ITA types) to examine the work for himself and offer his own considered judgment.
Um, no, Joshua, sorry, I read your post carefully the first time around, and your discussion just plain gets the research wrong. If you did in fact try to read the research carefully before writing, then I do apologize to you for inferring that you hadn’t; but I couldn’t see any better explanation for why, e.g., you used that patently inapplicable ice cream/drowning example; or why you failed to see, and apparently still fail to see, that the NEJM article is trying to make the causal case. I’ll defer to your own self-judgement as to whether you’ve committed the sin of sloppiness or just simply misunderstood the article in question. (Zywicki, however, is clearly being sloppy.)
We can start by considering your lead-off sentence: “There are two things that really disturb me in academia – poor correlation/causality conclusions and silly research. My morning news reading offered two good examples of each.” Given that your reading offered you no examples of either (or, at least, that these are not such examples), your caveat later about not blaming the journal, etc. would be completely insufficient, even if it were appropriate.
Which it isn’t. Your caveat would be perhaps minimally appropriate had the researchers in fact made only correlational claims, without making clear that that’s all they were doing, and the journalists had mistakenly drawn the causal conclusion. But that’s not the case here. In fact, the journalist didn’t make a bad inference to causation from the research, because the scientists successfully made the case for causation, as I thought the bits I quoted earlier made clear. So your caveat is simply besides the point.
I would add that I feel fairly justified in my having believed (even if it turns out mistakenly) that you hadn’t read the actual article before posting: you didn’t refer to any aspects of the research not mentioned in the short Indy Star article; you didn’t provide a link to the original study; and, as I said, your use of the example that you did really seemed to be evidence that you just hadn’t seen the significant passages in the article on controlling for confounds; also, the Zywicki piece is so completely off the mark about the IAT research that your using it as your key citation on that research indicated to me that you hadn’t checked out that research, either. I probably still shouldn’t have made the accusation in quite the way I did, because it’s somewhat inflammatory. (Though keep in mind that there’s some fairly inflammatory language in the original post (in particular the use of “silly” and the putting of scarequotes around “academic research” to refer to what are in fact two perfectly respectable pieces of research).) But if I drew the wrong inference about how much background reading you did, well, I do apologize for it, but I ask you to recognize that you’ve got a portion of the blame for that as well.
This “excellent job” consisted of accounting for the following variables, and only the following variables: age, highest educational degree, prior physical health, level of physical activity, use of vitamin E supplements, body-mass index, cigarette-smoking status, aspirin use, scores for the mental and energy health index, and degree of social integration. All other factors – birth weight, income, geography, other variables in standard of living, family history, genetics, other vitamin intakes, quality of friends (not just quantity), and countless other variables – were not considered. These are all vitally important factors to consider in cognitive ability and the effect that alcohol can have.
The NEJM is important in establishing a correlation, but it will take further scientific research to establish causation. That causation is likely present, though. Research is starting to suggest that alcohol appears to protect the brain in the same way that it guards the heart: by improving blood flow. But this is the research that will determine causation, not a correlation study. Journalists jump on correlation studies much too often as conclusive evidence.
I don’t at all want to deny that last claim, that journalists often make mistakes about correlation vs. causation. But we need to distinguish between making a tentative case for causation vs. simply observing a correlation. My contention is that Stampfer et al. are doing the former, and doing a pretty decent job of it. Of course there are other confounds one could check for, and it’s always a greater danger in a retrospective method (as opposed e.g. a randomized one) that a confound will slip by you. But if you’re only doing correlation, you don’t even need to begin to look for confounds, because correlation doesn’t care about confounds — to follow your earlier example, there’s nothing at all wrong in claiming an ice-cream/drowning correlation. That there are significant confounds is irrelevant to the correlationsal claim, as what a confound is is exactly a source of the correlation (but, i.e., one other than a causal relation between the correlated events or characteristics). So if Stampfer et al. were just doing a correlational study, then all the stuff about confounds, reverse causation, etc. would be irrelevant.
Now, if you want to make a case that they have not done a maximally good job of controlling for confounds, then that’s a perfectly good thing to argue about, though I’d guess we’re both insufficiently grounded in this particular literature to know which ones are worth looking at directly. Part of what one learns when one masters a given literature is what are good ‘usual suspects’ for such things, because you can never control for all possible confounds.
To just take one example from your list, I’d guess that Stampfer et al. had good reasons to check only on the vitamins they did and not others; or maybe others aren’t suspected of having an influence on cognitive function, or maybe other aren’t taken broadly enough in the population to explain away the effect. Also, some things that are measured directly are often taken as pretty good proxies for other things that aren’t; completed educational level, for example, is fairly well-correlated with things like standard of living and income. Also if geography has an effect presumably it would be by means of something other influence on on the Ss; e.g., geography surely influences diet, but that in turn influences other health factors that they did measure. (It also might influence drink of choice, which is something they did check for & which was shown to be irrelevant.)
But we need to be clear that this kind of critique is a critique of the case that is in fact being made for causation, and does not ground the initial charges that in this instance any confusion between causation & correlation was made. To argue about whether they sufficiently controlled for all plausible confounds is to argue about how good a causal case they have made. I still think it’s a pretty good one, though I certainly did not mean to imply that it’s an open-and-shut case or that we don’t still want to do tons more research. But it seems to me to be enough to draw reasonable health care decisions from, which is not something you can do if you’ve only demonstrated correlation.
Part of the problem here is that I think you may have committed a pretty sophisticated confusion in your last comment, between discovering a causal relation, and discovering the underlying mechanism of that causation. You’d be right in thinking that there’s nothing at all revealed by the Stampfer et al. about the underlying mechanism of how the alcohol consumption. But we can show the existence of causation without having first to show the how of the causation — indeed, one frequently shows the former before the latter, because we don’t think to look for the latter in the absences of good evidence for the former. To take a boring example: many of the causal properties of magnets were well-established long before anyone had a clue about the underlying physics.
So it still seems to me that neither the researchers nor the journalists have, in this instance anyway, committed a correlation/causation confusion.
Josh, occasionally I’ll describe the blogsophere as “Silicon Tower.” Although its not a perfect descriptor (yet), every once in a while my characterization is apt.