Hip Fractures: How to Be a Social Scientist, Sort Of

I recently was reading a health column by Jane Brody in The New York Times about people taking calcium supplements to prevent osteoporosis. As seems to be invariably true in her columns, Brody here took an anti-supplement position. She has some evidence: supplements don't seem to do any good in healthy women, and may do some harm, though not all the studies agree.

Brody's conclusion: Eat more dairy products.

The one indisputable fact is that the safest and probably the most effective source of calcium for strong bones and overall health is diet, not supplements. But few American adults, and a decreasing proportion of children and teenagers, consume enough dairy foods to get the recommended intakes of this essential mineral.

But I had some doubts. I had, for some reason, recently read this article from a medical journal, arguing that hip fractures (taken as an index of osteoporosis) were more common in countries that consumed more animal protein and less vegetable protein. (There was a theory behind this, something about "endogenous acid production consequent to the metabolism of animal proteins.")

This article hadn't specifically looked at dairy consumption, but had included a list of the 33 countries studied, ordered by incidence of hip fractures. Among the countries with the highest incidence were some that I thought probably had high dairy consumption, such as the Scandinavian countries. (Little-known fact: the Vikings were big dairy farmers.) Also, some countries with a strong cultural aversion to dairy products, such as China, had very low rates of hip fracture.

So I decided to investigate further. I found a source on dairy consumption worldwide. If I were doing this for a journal article, I wouldn't cite  a source that cited the FAO, I would just cite the FAO.  Using data on the country level rather than the individual level is not ideal, but no individual-level data was available. (I've decided to grit my teeth and treat "data" as a singular noun.) There were a few countries for which I couldn't find dairy consumption data, so I ended up with 28.

I then created an Excel spreadsheet and made a scatterplot of the data. (For complicated reasons,* I eventually used the natural logs of all the data, which is easy to do in Excel. That's what the "ln" on the axes means. The results are similar without doing that.) If dairy consumption reduced hip fractures, you'd expect to see the points form a rough line that sloped downward from left to right. Instead, the line slopes upward, meaning that higher dairy consumption is associated with more hip fractures. The correlation between (log of) dairy consumption and hip fractures is a strong +.91. (The chart will get bigger if you click on it.)

This certainly casts doubt on the claim that drinking milk reduces osteoporosis. But we're not done yet. Here's the hard part about doing social science: correlation just shows that two things tend occur together. The actual relation between those two things may be masked by other things. 

In this case, when we look at the countries that have low incidence of hip fractures, we see places like Nigeria, China, and Thailand. Countries at the high end include Sweden, Norway, and Germany. Perhaps it's higher income that leads to osteoporosis, and since people in higher-income countries drink more milk, we see milk consumption and osteoporosis together.

Is there a plausible theory that would explain why higher income could result in more osteoporosis? Yes: it's well-known that weight-bearing exercise reduces the risk of osteoporosis, and it's probable that people in poorer countries do more physical labor than people in rich countries.

So let's get some data on per capita GDP in these countries. We'll go to the CIA to get the information. Sure enough, the correlation between between (log of) income and hip fractures is strongly positive at +0.87,  and the correlation between income and dairy consumption is also strong at at +0.84. So it is possible that what we're seeing as an effect of dairy consumption is just the effect of higher income.

To go further, we must cross the line that separates bad social science from normal social science, and measure the effect of each variable holding the other constant. To do that, we'll use multiple regression, which is available on Excel by activating the Data Analysis pack. Here are the results


Look at the column "Coefficients". (Ignore the row labeled "Intercept.") This measures the effect of that variable on hip fractures. Roughly, the coefficients indicate the percentage increase in hip fractures for a one percent increase in dairy consumption or GDP per capita (hip fractures increase by 0.84% and 0.73%,respectively). But the important point is that both effects are positive, meaning that an increase in either is associated with an increase in hip fractures holding the other constant.

Now go to the column "P-value". Here we are asking the question, "How likely is it that the effect of each variable is real and not just a random fluctuation in the numbers we happened to collect?" The effect is probably real if these numbers are small; conventionally we want them to be .05 or smaller, and if so we say they are "statistically significant." As you can see, when we take account of GDP per capita, dairy consumption is statistically very significant (that second  number in the "P-value" column is .000098, which is really small), even though GDP is also significant.

But is this plausible? Do we have an explanation for why drinking milk should be bad for you?

There's only one explanation I can think of: the claim, in the paper I linked to above, that animal protein causes osteoporosis. Dairy is chock-full of animal protein. Let's see if dairy has an effect when we take account of the consumption of animal and vegetable protein, using the figures from the paper.

As you can see, consumption of animal protein (AP g/day) has a statistically significant effect on incidence of hip fractures, as does per capita GDP. But when we take account of those things, dairy has no statistically discernible effect. We have to be careful in interpreting this, though. It doesn't necessarily mean that consumption of dairy has no effect on incidence of hip fractures, just that it has no independent effect. It probably means that consumption of dairy has an effect through its effect on the level of animal protein consumed. And the effect is a bad one.

So it doesn't seem that Brody is on strong ground in recommending consumption of dairy products to ward off osteoporosis.

One last point: You may ask, what does all this have to do with social science? This is epidemiology. That's not a social science, is it?

Whether it is or not, the problems are the same: dealing with non-identical humans, and having to reach conclusions based, usually, on non-experimental data. Of course, there are a lot of natural sciences where you usually can't do experiments, but that's a subject for another time.

*Heteroscedasticity.