Why are We Basing Food Policy on Black Box Data?

Roller Coaster Numbers from Gates-backed Study Have Some Asking for Retraction

This is the Unsettled Science newsletter about why it’s so hard to get trustworthy advice on nutrition & health. Original, investigative reporting. Plus: What the science can actually say about optimal eating. By best-selling author and journalist, Nina Teicholz.

This news item is about a $380 million, influential study backed by the Gates Foundation, which is informing global health policies yet turns out to be producing highly unreliable numbers on the health risks of foods—red meat, especially. The study authors acknowledge that some numbers in a recent paper were inaccurate, yet they do not plan to correct or retract the paper.

To my knowledge, this topic has received no coverage in the US media.

Why are We Basing Food Policy on Black Box Data?

It turns out that a highly influential 2019 claim—that no amount of unprocessed red meat is safe for health—was completely inaccurate, according to a statement in March by the authors of the Global Burdens of Disease study (GBD), an on-going project funded by the Bill & Melinda Gates Foundation. Two years earlier, in 2017, these same authors had judged red meat to be the least likely cause of death among 15 risk factors analyzed. Then, in 2019, red-meat’s risk jumped 36-fold. A forthcoming publication will correct these errors, and the risk will drop significantly, said the lead author Christopher Murray, in an interview. Despite the inaccuracies, however, he says he does not intend to correct or retract the paper.

The problems came to light when an international group of scientists managed to get a letter printed in TheLancet, which publishes the GBD’s risk factor studies biannually. The letter alleged that Murray’s group had not considered important data contradicting their findings on meat and had, among other things, neglected to provide information on the methods of key scientific reviews. In laymen’s terms--they didn’t show their work. This meant Murray’s results could not be replicated by outsiders, a clear problem in science, where replication is considered a bedrock principle. GBD data methods are “locked black boxes,” warned a leading nutrition expert—back in 2013.

GBD isn’t just numbers. The study, with 7,000 researchers in 156 countries and some $380 million

¹

from the Gates Foundation, has long intended to be highly influential in the world of global health policy. “We very much want our studies to be used for policy,” emphasized Murray, who directs the Institute of Health Metrics and Evaluation (IHME) in Seattle, Washington.

Indeed, the 2019 paper, what IHME calls its “capstone publication,” has, as of this writing, been cited by 861 papers and 13 policy documents, including the UK National Food Policy--which states that red and processed meats are responsible for 29% of all diet-related deaths and disabilities, a statistic based on the flawed 2019 GBD paper.  

GBD has also been a collaborator with the World Health Organization since 2018, and its numbers are increasingly being used by the United Nations, including work to reduce meat consumption as part of the UN’s “Sustainable Development Goals.” The most prominent of these groups, EAT-Lancet, for which Murray was a “Commissioner,” aims for everyone on the planet to eat zero to 2.4% of calories as red meat.

Altering the world’s diet along these lines is intended to stop global warming, yet anyone can agree that global policy affecting human health ought to have a foundation in reliable data. With the still-rising epidemics of obesity and diabetes, we can’t afford false steps. In this light, GBD’s wildly fluctuating food-risk estimates look perilous.

GBD “cannot be a trusted source”

“Something seriously wrong with estimate” on red meat, tweeted one of the world’s top methodology experts, Gordon Guyatt, Professor in the Departments of Health Research Methods, Evidence and Impact and Medicine at McMaster University.

Salim Yusuf, distinguished professor of medicine, also from McMaster’s and one of the world’s most-cited experts, said to me in an email,

The 36-fold variation in the numbers of deaths attributed to red meat consumption…indicates how fragile the GBD estimates are and how changing a few assumptions in their calculations can dramatically alter estimates of the numbers of deaths from red meat.

It is very unlikely that new original data have become available  in the two years between the two publications, which can dramatically alter the results. Almost certainly, these differences are a result of different assumptions and methods. This shows that the GBD publications on red meat and likely on many other components of diet cannot be a trusted source of information to make policy recommendations.

In fact, other food risks calculated by GBD also changed dramatically from 2017 to 2019. The risk of salt dropped by 40%, while risks attributed to diets low in fruit, nuts and seeds, vegetables, seafood omega-3 fatty acids, and polyunsaturated fatty acids declined by more than 50%.

The letter to the Lancet was a wake-up call on these issues. It “raises serious questions about the methods and results of the GBD project and their estimates related to diet,” wrote Yusuf, although he also emphasized his belief that Murray is committed to good science and that collaborations with outside scientists in the future could lead to more robust results and conclusions.

The letter authors, led by Professor Alice Stanton of the Royal College of Surgeons in Ireland and professor Frederic Leroy of Vrije Universiteit in Brussels, said that given the instability and lack of documentation for the GBD data, it would be “highly inappropriate and imprudent” for any current or future policy or regulation to be based on this work. After Murray’s letter acknowledging inaccuracies was published, Leroy and other scientists called for the paper to be retracted.

Retraction? What the Guidelines Say…

The Lancet rarely issues retractions. As the world’s most influential general medical journal, it obviously takes these issues seriously, but the journal has deep ties to Murray’s group. In a partnership going back to 1990, The Lancet maintains a special area of its website for GBD. Since early 2021, Lancet journals have published some 25 GBD papers; for the 2012 version, The Lancet put out a 200+ page special triple issue, reportedly the first time in the journal's 189-year history that it devoted an entire issue to a single study.

Lancet Editor-in-Chief Richard Horton, explaining his commitment to partner with both GBD and EAT-Lancet (on whose advisory board Horton sits), called himself an “activist” on the environment and believes red meat should be seen as a luxury item, consumed only occasionally.

Horton may not be dispassionate on the issue, yet guidelines developed by the Committee on Publication Ethics, of which The Lancet is a member, state that a paper should be retracted if there is “clear evidence that the findings are unreliable…as a result of major error (eg, miscalculation or experimental error)…,” which is almost certainly the case here.

Stanton is an academic who has published extensively on preventive medicine and health promotion. She was also a clinician for 30 years and had been following GBD with concern, because its analyses excluded maternal and child health. For example, iron-deficiency and anemia seriously impair health in these populations yet are not measured by GBD. When she saw the 2019 GBD results, she felt compelled to write a letter. “I saw the change [in red-meat risk] and asked myself, how could they have done this? To me, this was a gross overreach, and I felt that we certainly needed to challenge it.”

Murray acknowledged to me that GBD does not consider the differing nutritional needs of growing children, adolescents, pregnant women, women of childbearing age, or older people. His models address only adults, 25 years and older, and examine only selected disease outcomes, not the specific nutrient needs for healthy growth or reproduction, he said. The potentially disease-preventing effects of red meat, as a source of high-quality protein and essential nutrients, including iron, as Stanton and colleagues describe in their letter, are not factored into the GBD model.

Letter Writers Struggle to Get Published

Initially, The Lancet editors rejected Stanton’s letter. She turned to the esteemed Annals of Internal Medicineand Journal of the American Medical Association but was rejected by both. The Annals Editor-in-Chief Christine Laine wrote in an email to Stanton that she agreed the group’s concerns were “valid,” and in fact the journal’s statistical editors had noted that “there are many more problems” with the GBD’s analytical approach yet declined to publish this evidence as it would be “inappropriate.” (For greater understanding on why Laine might be reluctant to publish anything on red meat, see this stunning piece in JAMA)

Six months after Stanton’s first submission, The Lancet reached back out to Stanton apologetically, one week after an item appeared on the matter in The Grocer. Another two months passed before the letter was finally published, in February.

Stanton’s group encountered another obstacle when Murray did not respond to multiple invitations from The Lancet to write a response to the letter. Declining is unusual, since authors are usually eager to respond to critics. Again,journalist inquiries to IHME including my own may have been galvanizing, such that when I spoke to Murray, he said he’d always planned to reply but that it had “slipped off our mind.”

Science Says: Show Your Work

In his response, Murray said the 2019 errors were due to the fact that his group’s modeling was undergoing a five-year reformulation that was not yet complete. He explained that his team’s analyses had mistakenly overlooked some findings on red meat being protective against hemorrhagic stroke, and they had not fully considered the unreliable nature of the evidence linking red meat to various other health outcomes.

In fact, the overall evidence to show any effect of red meat on any kind of disease is extremely weak, said Murray. When asked why his paper hadn’t considered any of the available evidence on this point, contradicting his alarmist red-meat findings, he said outside systematic reviews “weren’t that important,” since they tend to come to such varying conclusions.

Murray is correct on that score, and this is why elaborate standards have been set up--rules to follow, essentially—to ensure the rigor and transparency of scientific reviews. Roughly speaking, the closer one follows these rules, including the checklists PRISMA and GATHER, the more reliable the results. It was therefore a potential shortcoming of Murray’s own 2019 paper that its systematic reviews allegedly failed to meet any of the 27 items on PRISMA and  4 out of the 18 items GATHER, according to the letter-writers. The Lancet requires its papers to abide by both standards.

²

In his response to Stanton, Murray does not address these specific issues. He questions the usefulness of PRISMA and suggests GBD’s online “Data Sources Input Tool” is sufficient to comply with GATHER.

However, Murray does not deny the basic omission here, which is that GBD simply doesn’t publish its systematic reviews. For instance, we’re told the review on red meat looked at 92 papers but even the Data Sources Input Tool doesn’t tell a reader which ones. Normally, a systematic review will list the exact papers, methods for evaluation and the results in great detail.

“Locked black boxes” is how the leading English nutrition scientist W. Philip James described GBD methodology back in 2013, when Murray presented on GBD at a meeting of the London Royal Society, the UK’s equivalent of the U.S. National Academy of Sciences. There were “repeated calls for transparency and publication of the meta-analyses,” wrote James. “It was also stated – quite rightly, that no country could trust data and judgements of the study until its governments and policy-makers knew for sure where these actually were derived from…” These concerns are evidently enduring.

GBD “transfers power” to the “richest and most powerful” groups in public health

Challenges to GBD have been few, probably owing to the fact that the study and its backers are so extraordinarily wealthy and influential. As two researchers from Johns Hopkins University wrote in a 2020 analysis of GBD, its ability “to self-correct is likely to be hampered,” because opposing viewpoints cannot be heard on an “uneven playing field.” While the IHME “is assured publication of its data in the world’s most widely read journal publishing on global health,” its critics, by contrast, “operate with little funding and make their arguments in specialised outlets…with limited circulation and reaching largely only academic audiences.” The uphill struggle faced by Stanton and her co-authors in getting their letter published could be seen as an example of this dilemma.

The fact that the GBD study transfers power from low-income countries and the disempowered to some of the “richest and most powerful organizations in the global health field,” such as the Gates Foundation, is “arguably…the deepest concern,” the Johns Hopkins authors write.

Potential conflicts of interest cannot be ignored, they add. The Lancet, for example, is “operating in a competitive medical journal market and owned by a large for-profit publishing company, might have incentives that extend beyond knowledge production.” Indeed, the Lancet’s “impact factor,” a prized measure of a journal’s influence, increased by 8-9% after launching the IHME publications.

The Lancet’s Richard Horton may also have a financial conflict of interest after receiving the $100,000 “Roux” prize in 2019, which is funded by David Roux, a founding board member of IHME. Murray said the prize was “not in the slightest” a conflict of interest, because IHME had “no role in picking that prize,” yet the website says IHME’s Board Executive Committee selected the honoree.

For global health experts, the question will be whether the still-evolving methodology of GBD, the lack of documentation or sharp fluctuations in risk estimates will cause policy makers to hit the pause button on using these data as a foundation for reliable population-wide public health recommendations.

Note: As with all my work, I appreciate comments, suggestions, and corrections. Verifiable corrections will always be made asap.

Piece updated on 5/15, with the following: 1) It is more accurate to say that Murray did not reply to multiple requests by The Lancet to respond to the letter writers. Previously I had written, based on a tweet, that Murray declined multiple requests. I was subsequently able to see the original correspondence on this issue from The Lancet. 2) David Roux’s name was added to the piece, and he is identified as a founding board member, not a founding member.

---

1. $105M at its founding in 2007 + $279M more recently:
Paulson T. $105 million Gates' gift helps start global health center. Seattle PI Reporter, 4 June 2007. http://www.seattlepi.com/local/article/105-million-Gates-gift-helps-start-global-1239473.php#ixzz2H3FMx2ja.
More recent funding: https://www.healthdata.org/about/history

2. Confirmed in an email from the Lancet