Confounders, Biases, and the Sweetener Science

More on the potential pathology of nutrition science

Confounders, Biases, and the Sweetener Science

One of the old dictums of journalism, old even when I was young, is that dog biting man is not news, but man biting dog is. The former is what we expect to happen. The latter is unusual, the opposite of our expectation and so, in a word, newsworthy.

In the personal health sphere, a study that shows you can lose or maintain your weight using artificial, non-caloric sweeteners is also not news. That’s why these sweeteners exist, isn’t it, as a means of weight control? But if researchers report the opposite, that these non-caloric sweeteners actually make you fatter (or cause any disease at all), that’s going to end up in the news. And it has.

Yet another seminal epidemiologic study has linked artificial sweeteners to fat accumulation, leaving us, once again, with the kind of conflict that plagues virtually all of nutrition science. This conflict is not only between what’s news and what’s not, but what epidemiologists would like to believe is true — that these diet-disease correlations imply causality — and what the experimental tests repeatedly suggest is not.

As even the World Health Organization (WHO) acknowledges, artificial (i.e., noncaloric) sweeteners do indeed seem to help with weight loss in clinical trials. The caveat is those trials tend only to last a few months. What about longer-term use? We don’t use these sweeteners for just a few months. We use them throughout our lives. That’s where the epidemiology comes in. When people who use these sweeteners for years to decades are compared to people who don’t, the ones who come out the fatter are the users, suggesting that sweeteners do more harm than good. It’s this kind of long-term comparison that’s the essence of the new report. And, indeed, even the authors suggest that their findings are sufficiently worrisome that we should rethink national recommendations to replace added sugar with artificial sweeteners. In a word, seriously?

Before we work through the answer to that simple question, allow me to digress on the broader context.

The complacency problem

My obsession with the quality of public health research began in the early 1990s with an article I wrote for The Atlantic, “Fields of Fear,” on what my fellow journalists had turned into the anxiety of the era: the notion that electromagnetic fields, EMF, from power lines, were carcinogenic. The primary evidence then was epidemiological: studies reporting that greater exposure to EMF was associated with greater prevalence of certain cancers. In this case, both the researchers and the journalists, most notably, Paul Brodeur, writing in The New Yorker, assumed this association was likely enough to be causal that they could widely disseminate the anxiety that EMF was carcinogenic.

What fascinated me in the course of my reporting was how little the epidemiologists I interviewed seemed to care about the possibility that they were overinterpreting their evidence, or as the Nobel Laureate physicist Richard Feynman might have put it, that they were fooling themselves. I had spent much of the previous decade being schooled by chemists and experimental physicists about how unimaginably easy this was to do, this misinterpretation or overinterpretation of your evidence, and how rigorous, methodical, and skeptical researchers had to be to minimize this possibility. This is what makes scientific endeavors so extraordinarily, if not, occasionally, impossibly challenging: trying to establish beyond reasonable doubt that you, the researcher, can trust your interpretation of the evidence. And here were the epidemiologists, with a very few notable exceptions, seeming not to care.

Their argument, in a nutshell, was that they were doing the best they could, and so (in the interests of public health) they had an obligation to assume that their best was good enough.  They knew that the associations observed in their studies were only “hypothesis-generating,” as they would tell me. But they also knew that they couldn’t test these hypotheses with experiments. Nobody was going to do a trial, randomizing participants to high or low EMF fields, and then following them for decades to see which group might have the greater cancer incidence. And because these epidemiologists considered their hypotheses so important, they were going to believe their ideas were very likely to be true. Hence, the concept of prudent avoidance was evoked: they didn’t know if their interpretation of the data was right or wrong, but they were going to act as though they were right, and then disseminate that belief, widely and with confidence. Then we could avoid living near electric powerlines just in case they were right. This would be the prudent thing to do. An obvious and extreme example of this prudent avoidance philosophy was the Covid shutdowns of 2020.

What I didn’t know then (and should have) is that the Yale epidemiologist Alvan Feinstein had already famously called out the discipline of epidemiology in a series of critical articles, most famously one in Science in 1988, “Scientific Standards in Epidemiologic Studies of the Menace of Daily Life.” He accused his colleagues in the field of perpetrating what he called “apparent complacency about fundamental methodologic flaws.” And that’s what I had witnessed. Feinstein himself would be attacked for taking money from the tobacco industry, but this didn’t mean that his critiques did not have considerable merit.

The field of chronic disease epidemiology was relatively new at the time, specifically as a tool to identify what Feinstein had called “the menace of daily life” – diet, lifestyle, pharmaceuticals, and environmental exposures, like EMF. The field emerged out of the great triumph of the endeavor: the observation that cigarette smoking caused lung cancer. That, too, was an association, but it was believably causal because the link between smoking and lung cancer is enormous: a 10-20x increased risk. We cannot imagine how to explain such a powerful association by any other combination of biases or confounding factors. So we accept that it is true.

But what about all the other potential menaces of daily life that might cause disease? EMF, for instance, or artificial sweeteners. The associations observed are far smaller, typically a tenth the size of the link between smoking and lung cancer.  The diseases to which these menaces are linked are far more common than lung cancer (in non-smokers). The exposures are not binary – smoking vs. not smoking – and they’re difficult to assess. More importantly, exposure is also dependent on or associated with a host of mostly unknown factors – confounders, in the lingo – that might also influence disease risk.  Health consciousness is one of these confounders, most notably and as we’ll discuss, as is socio-economic status, which associates with it. Feinstein’s point was that we cannot assume these other daily-life-menace associations are causal, as we do with smoking and lung cancer, because we can imagine all too many reasons why they might not be, why this kind of evidence fools us. Epidemiologists will pay lip service to these problems. They will toss them in a bucket that they call “residual confounding,” which can be defined as all the possible reasons that they might be fooling themselves, but then they will dismiss them as unlikely to explain the associations reported. And what we want to know is whether this assumption that these confounders and biases do not explain the observed associations is ever valid.

The Nurses Health Study and the lesson not entirely learned

The first lesson that associations may be tragically misleading came out of the seminal Nurses Health Study (NHS), which was founded in 1976 by Harvard epidemiologists with an enormous cohort of 120,000 nurses. In 1985, the NHS researchers published their first controversial and significant finding – that postmenopausal hormone replacement therapy (HRT) “reduces the risk” of severe coronary heart disease by 50 to 70 percent, a dramatic, remarkable reduction. The implication was that HRT should be prescribed to menopausal or post-menopausal women to prevent heart disease, as it would be.

Because the HRT-heart disease association was so dramatic, a 50 to 70 percent reduction in risk, it prompted other researchers running these kinds of epidemiologic surveys to look into their data sets to see if they had seen it, too. They did, but they also observed similar associations between HRT and other health endpoints that seemingly had little to do with HRT itself: reductions in suicides, for instance, and deaths by violence. If all these reductions were related, then HRT use was not the cause. It had to be something that itself associated with both HRT use and these other endpoints as well. Beginning in 1987,  these other researchers, began to suggest ways, obvious ways, that the Harvard epidemiologists might have misinterpreted their data. The gist of it was that maybe women who used HRT were different in many ways from women who didn’t. Despite these women all working as nurses, those using HRT could still be wealthier, have a higher socioeconomic status, for instance, or better doctors. They might even be healthier to begin with. And maybe all these subtle inequalities could be added up to explain the disparities in health.

The only way to know for sure, of course, was to test the hypothesis generated by the HRT-heart disease association. The only reliable way to do that was in placebo-controlled randomized trials. These would (mostly) assure, through the placebo control and the randomization, that the only meaningful difference between HRT users and non-users was the HRT itself. When such trials were completed – most notably, the Women’s Health Initiative-- it was clear that the Harvard epidemiologists had fooled themselves. They had overinterpreted their data: they had assumed that their survey of over 100,000 nurses could distinguish a causal association from a noncausal one but in fact, it could not. And it could not do so, even with something so relatively easy to quantify as whether or not these women had ever been on hormone replacement therapy.

Yet because the results of the WHI weren’t published until 2002, the epidemiologic community had another decade and a half to commit to using these prospective cohorts to do something they might not be able to do: to train students in epidemiology, to get grants to continue doing so, and to publish (tens of?) thousands of articles implying that the associations they observed might reliably predict causality. These studies then became fodder not just for the latest news reports on health, but for the recommendations and guidelines issued by public health and medical organizations. Epidemiological studies had taken over these disciplines not because their evidence could be trusted to produce reliable knowledge, but because they were relatively easy and relatively inexpensive to do, and epidemiologists built their entire careers on the creation and use of these prospective cohorts. Moreover, once the evidence was gathered, reliable or not, anyone with access could play the game of trying to identify a menace of daily life.

The situation reminds me of a quote from Edith Nesbit and her children’s book, The Magic City, published in 1910: “..there's a dreadful law here—it was made by mistake, but there it is—that if anyone asks for machinery they have to have it and keep on using it.” It doesn’t matter in this case whether the machinery can do what the epidemiologists would like it to do, they have it and they have to keep on using it. Just as researchers can fall in love with a hypothesis and bias all that comes after, they can commit themselves to the use of an unreliable method of accumulating evidence and do just as much damage. And here’s where Feinstein’s accusation of methodologic complacency comes in: they may not try very hard to establish that their evidence-gathering machinery and their methodology can’t do what they’d like it to do, because that would put them out of business. The bias is unavoidable.

If this were a poker game, the epidemiologists would be accused of throwing good money after bad. Of playing a bad hand and refusing to give it up. Folding the hand was never an option for these people, because it’s the hand they dealt themselves, and it’s the only one they’re going to get.

CARDIA and the Sweetener Story

Now back to artificial sweeteners: The epidemiologic survey that has provided us with the latest artificial sweetener anxiety is known as the CARDIA (Coronary Artery Risk Development in Young Adults) study. It was launched in the mid-1980s to “examine the distribution of [cardiovascular disease (CVD) risk factors (RF)] in young adults; to identify associated lifestyle, psychosocial, and other factors; and to assess longitudinal RF evolution in early adulthood.” CARDIA researchers recruited over 5,000 young participants -- black and white, men and women, on average 25 years old -- in four major cities. These participants were questioned about their diet and lifestyle when the study started and then again 7 and 20 years later They were subjected to regular physical exams. The result has been hundreds of articles reporting on the associations observed.

This is all very impressive, but when CARDIA was launched, it built in the biases and inadequacies of the era. The CARDIA researchers, most notably, had yet to have the benefit of the Nurses Health Study’s learning experience. By the time they did, it may have been too late. They were committed to a survey that could be useful for associating biological risk factors with disease states – blood pressure, say, and hypertension – but might be hopelessly misleading for the purpose of identifying the lifestyle or dietary factors that might be causing those conditions.  

Now we have the CARDIA researchers suggesting that artificial sweeteners are, in a word, fattening. The researchers found an association between long-term use of artificial sweeteners, particularly aspartame and saccharin, and greater volumes of visceral (in the organs), intermuscular, and subcutaneous adipose tissue. Hence, these artificial sweeteners may cause fat accumulation, and exacerbate the chronic diseases associated with fat accumulation.

One important point that the CARDIA authors readily acknowledge, as I mentioned, is that artificial sweeteners have been tested in clinical trials to see if they help with weight loss, and they apparently do. And this is the opposite of what the CARDIA associations themselves apparently suggest. Quoting the paper (my italics), “Contrary to our findings and those of other observational studies, RCTs [randomized controlled trials] of ArtSw [artificial sweeteners] have demonstrated modest but inconsistent weight loss effects, which are likely due to controlled or reduced caloric intake among the individuals participating in these trials.” In short, when the hypothesis generated by these associations was tested, it mostly failed the tests. This is a common, if not the common result of such clinical trials. The epidemiologists prefer to blame the tests, the randomized-controlled trials, for not properly testing their hypotheses: Perhaps the trials didn’t last long enough, or they randomized the wrong population (the excuse for the HRT conflict). The epidemiologists will not be complacent about pointing out the potential shortcomings of these trials; they are, in effect, defending their own research. We, as observers, can be more objective, because we don’t put ourselves out of business by doing so.

The numbers

Let’s start by assessing the numbers themselves. Those are typically left out of the news reports and subsequent discussions, but they are, as they always are, critically important. How big of an effect are we talking about here? If the use of artificial sweeteners is going to make me fatter over the course of 25 years, how much fatter am I likely to get?  The relevant data are in Table 2.

The researchers divided up the participants into quintiles of artificial sweetener use: the fifth of the participants who used the least artificial sweeteners to the fifth who used the most.

One observation that immediately stands out is that the first four quintiles go from using virtually no artificial sweeteners at all in quintile 1 – a mean of 22 mg/day, which is the sweetener content of a few ounces of Diet Coke – to a mean of 83 mg/day in the fourth quintile. That’s almost three times as much, but still the equivalent of less than half a 12-ounce can of Diet Coke (200 mgs) a day. The usage in quintile 5, in comparison, explodes to almost 400 mg/day. That’s almost five times as much as the next lowest quintile and almost 20 times what quintile 1 consumed. It’s the equivalent of two 12-oz cans of Diet Coke a day. If the folks in the last quintile were sweetening their coffee or the sweets they were baking, this would be the equivalent of a dozen packets of saccharine or aspartame a day or over 30 packets of sucralose, known as Splenda.

So what’s going on with quintile 5? Why are they so willing to consume this stuff when the others are either using very little or abstaining entirely? Recall the hormone replacement therapy story: We had to ask whether women who used  HRT and women who didn’t differ in meaningful ways other than the HRT itself. Here we have to ask whether the only meaningful difference between folks consuming the equivalent of two Diet Cokes a day worth of sweeteners (or a dozen or more packets in their coffee or baking), and those consuming essentially none is exclusively the amount of sweetener. And, of course, by meaningful, I mean capable of having some effect, directly or indirectly, on their fat accumulation.

Before I address that, let’s look at the variable to which these quintiles are associated: fat accumulation. These numbers are also little different between quintiles 1 and 4 – those who either abstain or use very little -- and they only get significant with quintile 5. Visceral fat (VAT in Table 2), for instance, goes from 128 to 130 milliliters in quintiles 1 to 4, which is 2 extra milliliters of fat accumulated over 25 years. Then, the number jumps up to 139 ml in quintile 5. Intramuscular fat (IMAT) goes from 2.25 to 2.33 ml in quintiles 1-4 and then rises to 2.55 in the fifth. (Curiously, VAT, IMAT, and subcutaneous fat are all higher in quintile 3 than 4.) BMI increases from 29.4 to 30.4 in the first four quintiles –from just under the obesity threshold to just over it -- and then to 31.6 in the fifth.

These are infinitesimal to small differences anyway, but it’s clear that the folks in quintile 5, with their 2-Diet-Cokes-a-day habit (or their dozen-plus packets of sweetener), are driving the associations.

Finally, how much did body weight itself change? Over the course of 25 years, the folks in quintile 1 gained 15.6 kilograms, a little over 34 pounds. That increases in quintile 4 to 17 kg, or 37 pounds. This is a difference of three pounds or less than 10% of all the weight gained anyway. In quintile 5, they add another 1.2 kilograms, another 2.5 pounds.

Here’s the point: all these initially young men and women got heavier (on average) by at least 34 pounds over the quarter century covered by this CARDIA analysis. Yes, quintile 4 got three pounds heavier still, and quintile 5 another 2.5 pounds, but that’s just a small proportion of all the weight they gained, whether they used artificial sweeteners or not. Waist circumference is even more telling: it increases by 16.4 centimeters in quintile 1; 16.8 in quintile 3, and then 17.6 and 18.1 cm. in quintiles 4 and 5  respectively.

Translating this to inches, even the folks who consumed essentially no artificial sweeteners saw their waist circumferences increase by 6.5 inches, and the ones who consumed the equivalent of 2 Diet Cokes a day saw it increase by a little more than 7. In short, if we started with two participants of identical size and weight, one in quintile 1 and one in quintile 5, and they stayed in those quintiles for the next 25 years, at the end of that time, they’d both be buying the same larger size belt, and they’d both be putting the prong in the same hole, but the belt would be a smidgen tighter for the participant in quintile 5 than for those in quintile 1.

What this suggests is that, if nothing else, 80 to 90+ percent of the fat these people accumulated had nothing to do with artificial sweetener use. And the entire discussion and any public health guidelines that maybe we should avoid these sweeteners is about the remaining 10 to 20 percent. The extra two-thirds of an inch of waist circumference over 25 years, the extra five pounds on top of the 34 we’d gain anyway.

Imagining how to fool an epidemiologist

Considering how small this weight gain is, can we even pretend to be confident that the people consuming the most sweeteners (quintile 5) can be compared in any way to the folks who are consuming very little to none at all? Imagine you have two friends. One avoids artificial sweeteners (hint, perhaps because they think of them as artificial and so, chemicals that might be dangerous), and another, the same size and weight, drinks multiple Diet Cokes each day or perhaps pours packets of sweeteners into his, her or their coffee. Can you really imagine that these two people are identical in every other way that might have some influence on their weight? (Looking at Table 1, about characteristics at baseline, the answer to this question gets even more obvious: the folks in quintile 5 seemingly consumed almost twice as many calories per day as those in quintile 1.)

The CARDIA researchers talk about “adjusting” for the potential confounders of diet and lifestyle and even physiology, e.g., age, sex, BMI, alcohol consumption, and physical activity, such that they can believe even the tiny effects seen in their study are likely caused by the sweeteners. Should we share their confidence?

One obvious explanation for the association observed is known as reverse causation: i.e., it isn’t that artificial sweeteners make people fatter but that people who struggle with their weight are more likely to use artificial sweeteners than those who don’t. Hence, the causation is reversed: having a weight problem causes you to use artificial sweeteners, not vice versa. That’s always a possible explanation for this type of finding. The authors say “the potential for reverse causality is unlikely but remains possible” and they cite a 2020 article that discusses the necessity of acknowledging in these studies the potential for reverse causality, albeit not why reverse causation might be unlikely.

The authors suggest they can assume reverse causation plays no significant role in the association they observed because they “adjust” for how much these people reported eating in the diet surveys and compare their diets to what the American Heart Association considered a healthy diet circa 2015. The researchers also say that the long follow-up period – 25 years – somehow “reduces the probability of reverse causality.” This is the epidemiologic equivalent of wishful thinking. It would be nice if they cited a reference to justify this conclusion, but they don’t.

More importantly, what they say in the paper is not actually what they mean. What they mean is that the long duration of the study reduces the likelihood that reverse causality can explain everything that they saw, the entire very small associations reported.  And the point is: it doesn’t have to. Maybe it explains some of the association, and the rest is explained by other biases or confounders built into this type of epidemiological study? Such as the residual confounders we discussed earlier.

The inevitability of residual confounding

Again, keep in mind that the associations between artificial sweeteners and fat accumulation are tiny. And maybe reverse causality explains some of what’s seen, but not all. This is where this concept of residual confounding becomes relevant. In a 2022 report on artificial sweeteners by the World Health Organization, the WHO authorities merely say that “further research is needed to determine whether the observed associations [not just to obesity, but type 2 diabetes and heart disease] are genuine or a result of reverse causation and/or residual confounding.” That “and/or” is critical. What we could be seeing with these tiny effects are the results of numerous small confounders and biases (i.e., misleading associations), all conspiring together to create this mirage of an association between fat accumulation and sweetener use. Once again, the epidemiologists are essentially defining residual confounding as all those factors that associate with artificial sweetener use, and that might directly or indirectly influence disease risk, but that they cannot understand or properly assess.

The most obvious of these are encapsulated in the concept of the healthy-user problem. It’s likely to be a major confounder at work in all these prospective epidemiologic studies.

The idea is that already by the late 1970s, a conventional wisdom existed about the nature of a healthy diet and the menaces of everyday life when these epidemiologic surveys were being established and the first diet assessments taken. Hence, the people who adhered to that wisdom – who ate, say, a Mediterranean-type diet (as in the MIND study discussed in my last post) or a low-fat, low-salt diet – could be expected to be the folks who paid attention to what authorities told them to eat. These people weren’t eating a Mediterranean diet because their parents or grandparents came from Crete or Majorca, but because they were consumers of health information and that’s what the authorities were telling them was the healthiest way to eat. In short, they were health conscious, and they could afford to be health-conscious (to eat what they thought was healthy, whether or not it was the least expensive way to eat) and, as a corollary, they had a higher socio-economic status, better doctors, greater education, etc., than those who didn’t follow these supposedly healthy dietary patterns. The health-conscious ones may even have been healthier to begin with and so worked harder to maintain that health status. I would bet that if they went out of their way to eat a healthy diet, they also engaged in other health-conscious behaviors. They were, in effect, the healthy users of the advice being disseminated by the (supposed) experts.

All of these advantages would contribute to better health over the duration of these prospective studies, independent of the diets they ate and the sweeteners they used, and even the amount of exercise they took, and these advantages might explain much, if not all of the associations observed in these studies. The epidemiologists could speculate that they might be able to correct for these confounders using ever-more sophisticated statistical adjustments to do so, but they’d always be guessing. Could they really adjust for the organic vegetables and personal trainer enjoyed by the health-conscious person? Let’s just say, I’m skeptical. Although “of course not” would also be an appropriate response.

The healthy user problem with a twist (i.e., confounding the confounder)

Part of the guesswork involved in assessing these healthy user biases would be in establishing not just how much they might contribute to any one reported association, but even which direction the biases go. Do they make it more or less likely that the association observed was a causal one--i.e., do they all go in the same direction?

Short answer: not necessarily. When we’re talking about the potential dangers of artificial sweetener use, as with any food, the question is always, compared to what? Are we comparing artificial sweeteners to sugar? Is drinking 2 cans a day of Diet Coke better for your health than drinking 2 cans of regular Coke? More or less fattening (independent of the calories in the sugar)? Or are we comparing it to drinking no sweeteners at all, caloric or otherwise? Is drinking 2 cans daily of Diet Coke better for your health than not drinking any?

We can imagine that folks who prefer artificial sweeteners to sugar are folks who are more health conscious, and their healthy user bias would favor the artificial sweeteners. We can also imagine that in this comparison, reverse causality would apply: the more you struggle with your weight, the more likely you’re going to use artificial sweeteners in lieu of sugar. It may or may not be true. Both, though, are reasonable assumptions.

But the comparison in this latest CARDIA analysis is not between people who use artificial sweeteners and people who use sugar or high fructose corn syrup. It’s between people who use artificial sweeteners and people who don’t. It’s not between Diet Coke drinkers and Coke drinkers. It’s between Diet Coke drinkers and water drinkers. Or people who put Sweet ‘N Low or Equal in their coffee vs. people who don’t. We have no idea how much sugar or high fructose corn syrup these people used. It’s not reported or discussed in the paper.

When you’re comparing people who use artificial sweeteners to people who don’t, I’m guessing the healthy user bias favors the latter: the abstainers. I think most of us who don’t use artificial sweeteners, avoid them because we think of them as chemical additives and so, bad for us.  Avoiding artificial sweeteners or minimizing their use could be what epidemiologists would call a marker of health-conscious behavior. It’s a sign of someone who’s health conscious in other ways, too. In short, someone who drinks two Diet Cokes a day is likely different in many ways from someone who drinks bottled water, and the aspartame in the Diet Cokes may very well be the least of it.

Now we have two possible explanations for the tiny associations observed with artificial sweetener use. One is that some people using artificial sweeteners are doing so because they’re struggling with their weight anyway, hence reverse causality is at play for them. The other is that some of the people minimizing their artificial sweetener use are doing so because they’re health conscious and don’t want to consume chemicals. The whole galaxy of phenomena associated with health consciousness– social, economic, physical, and otherwise -- is being mistaken for the causal effects of the sweeteners themselves.

Is this a playable game (i.e., should this hand have been folded)?

Given enough time, we could probably imagine dozens of other possible non-causal explanations (confounders and biases) for effects as small as these. Such an exercise would not be an excuse to avoid paying attention to an association we find inconvenient; it would be what scientists are supposed to do, their moral and professional obligation. It would be what is an absolute requirement for understanding the reliability of the evidence they have gathered. You would think it would be what any curious epidemiologist would have done because they would have wanted to know if they were fooling themselves.

In an ideal world, we would have had learned committees of these epidemiologists, working with biostatisticians and whatever other specialists are necessary, to rigorously assess every imaginable confounder and bias in these studies. They would do what’s necessary to arrive at a reliable estimate of the influence of as-yet unimaginable confounders (the unknown unknowns), impossible as that may sound, and they would set ground rules about when associations really can be considered sufficiently unlikely that the epidemiologists can even vaguely suggest that their observed associations might be causal. (And so publish their papers.)

But this brings us back to my poker analogy: maybe the epidemiologists should have just folded their hand and walked from away the table. Such an exercise in trying to assess and understand this universe of residual confounders and all the ways that these epidemiologic surveys might be unreliable, and then trying to quantify these effects in a way that would translate to each individual survey, would generate an endless accumulation of untested assumptions. Each of these, in turn, would have to be tested in clinical trials to see if they hold up. The exercise would be inordinately expensive and might take decades or longer to complete. It’s also possible, if not perhaps likely, that the epidemiologists themselves cannot be trusted to do such an obligatory exercise in good science themselves, because they’re so invested in the use of these epidemiologic surveys that their livelihood and their reputations and even their self-images depend on it. 

The simple alternative and what I consider the safest bet is to assume that the randomized controlled trials, when they exist, get it mostly right. Yes, even those trials have to be examined to make sure they are interpreted correctly and that they were designed without biases that might influence their results. But at least the controlled nature of these experiments gives us some hope that if we see an effect, it’s due to the substance or variable being tested and not all these other potential confounders. If no such clinical trials exist, then I’d wager that the associations observed – again, with a precious few striking exceptions like cigarettes and lung cancer – are the results of biases and confounders built into these surveys, not causal as the investigators are always hoping. If the title of a paper says something you eat or drink associates with or is related to or is linked to some disease, say to yourself, “Hmm, that’s an interesting hypothesis, I wonder what would happen if they’d actually test it. I wonder if they already have…” Then believe the results of the test.

Last note. Before I moved out of New York City in 2010, I had lunch with an old friend at The New York Times who was one of the paper’s best medical reporters (and, no, not Gina Kolata). We were discussing what kind of caveat could be put at the top of any article reporting on the associations observed in these epidemiological surveys such that the reader would know what to expect. One problem with newspaper journalism is usually the caveats go at the end (inverse triangle style, in the journalism lingo) and by then, the impact, if the reader gets to the caveat at all, is minimized. My friend, who had an iron-clad policy never to report on these studies, suggested that every story should begin with the words, “Everything you’re about to read may be bullshit and probably is.” I think she nailed it.