US Dietary Guidelines: Science is "Subpar," Cannot be Replicated

New paper finds systematic reviews to be of “critically low quality"

A crucial feature of any scientific finding is that it can be replicated. Failure to replicate is an ongoing crisis in science, made famous in 2005 by John Ioannidis, an evidence-based medicine expert at Stanford University, with a seminal paper in PLOS Medicine, "Why Most Published Research Findings Are False." Since then, a project on 50 landmark cancer biology studies managed to fully or partially replicate only 50% of them. Another large-scale project attempting to reproduce the significant findings of 100 studies from top psychology journals could do so for only 36%.

Still, we tend to believe that our national nutrition policy, the U.S. Dietary Guidelines for Americans, must be sound. After all, it’s been the foundation of our ideas about healthy eating for 45 years; many Americans don’t even remember a time when the guidelines weren’t in existence. Issued by the U.S. Departments of Agriculture and Health and Human Services (USDA-HHS), this policy is updated every five years, with a new edition (2025-2030) due out this year.

Now, a paper in the prestigious American Journal of Clinical Nutrition (AJCN) reports that the systematic reviews (SRs) underpinning the current guidelines (2020-2025) are “suboptimal” and of “critically low quality.”(Full disclosure: the non-profit I founded, The Nutrition Coalition, funded this paper)¹

The authors, led by University of Ottawa methodology expert David Moher, aimed to assess the reliability and reproducibility of these reviews. Fellow authors include others from Ottawa and from Dalhousie University, the Hong Kong Baptist University, and Baylor College of Medicine in Houston. (Because the paper’s lead author is Alexandra M. Bodnaruc, I’ll refer to it as Bodnaruc et al.)

They selected eight SRs on the guidelines’ key recommendations, specifically, the USDA dietary “patterns” that emphasize fruits, vegetables, whole grains, nuts, seeds, legumes (peas, beans, lentils), seafood, eggs, lean meat, and low-fat dairy. This basic advice has remained virtually unchanged since the first guidelines were put into place in 1980.

The first step of any SR is to do a literature search, where relevant papers are culled from scientific databases. When Bodnauruc et al. tried to replicate the USDA searches, they could not.² They write, “[W]e identified several errors and inconsistencies in the [USDA]³ search strategy and could not reproduce searches within a 10% margin of the original results” (10% is considered generally acceptable).

The result was, in fact, much worse than Bodnaruc et al. report. Their reproduced search yielded 10,201 papers compared to just 3,550 found by the USDA group (duplicates were removed in both searches), a three-fold difference. This enormous discrepancy alone ought to cast serious doubts on the USDA reviews.

In attempting to explain the difference, Bodnaruc et al. write that “there were errors and inconsistencies in the search strategy [of the USDA reviews] that would affect the retrieved results,” with the “most notable issues [being] the lack of quotation marks when searching multiword terms, as well as missing field codes for title and abstract…” (such strategies are essential to a database search on any research topic). They also criticize the USDA’s decision to exclude specific publication types at the outset of their reviews (editorials, comments, retracted publication, retraction notices, and systematic and narrative reviews), “because it can lead to excluding potentially relevant records.”

Bodnaruc et al. document other shortcomings in the USDA systematic reviews:

• The USDA did not conduct quantitative meta-analyses, having decided a priori to do “narrative” reviews, which are considered less rigorous and far less reliable.

• The protocols were not pre-registered with an independent group (e.g., Open Science Framework), a “best practice” that “is associated with significantly higher SR methodological quality and reporting transparency.”

  • It’s worth noting here that the USDA did post its protocols on its own website, but in my decade of analyzing the guidelines, I’ve found that USDA website pages often disappear, even from the Internet Archive Wayback machine, where they ought to be permanent.

• The SR process lacked transparency; the USDA team did “not appear to use…open materials, open data, open code, or open peer review.”

• The USDA team neglected to use or complete the standard checklists (“PRISMA,” “AMSTAR,” and “SWiM”) required of all high-quality SRs to ensure rigor and transparency.

On the bright side, Bodnauruc et al. found that the SRs met 12 other methodologic criteria, including no evidence of “spin bias.”

Still, this overall lack of a transparent and rigorous methodology exposes the process to potential bias. This point was underscored for me by Chirag Patel, an associate professor of biomedical informatics at Harvard, who wrote in an email:

Currently, there are many places in the [U.S. Dietary Guidelines] systematic review process where the possibility of bias can influence the results and ultimately limit the generalizability of findings, often calling into question the recommendations.

USDA-HHS Call Criticism “Misinformation”

The USDA can’t plead ignorance on this one. A 2017 report by the National Academies of Sciences, Engineering, and Medicine (NASEM)—mandated by Congress with a $1 million authorization—noted that the USDA’s methodology for reviewing the science “falls short of meeting the best practices for conducting systematic reviews” and “require[s] increased rigor to better meet current standards of practice.” “To develop a trustworthy [guidelines],” said the report, “the process needs to be redesigned.” The NASEM ultimately made 11 recommendations, but the USDA fully adopted none.

The USDA is also aware of a 2022 paper in PNAS Nexus (a NASEM journal) that I co-authored with several former members of the Dietary Guidelines Advisory Committee. We found “a continued lack of a fully rigorous scientific process for producing consistent and trustworthy guidelines for the public.” USDA-HHS officials responded to our claims last year in AJCN with a paper entitled, “Addressing Misinformation About the Dietary Guidelines for Americans.”

This paper from Bodnauruc et al. confirms that the USDA has not upgraded its systematic review process. And so, as NASEM sees it, the dietary guidelines recommended for all Americans are still not “trustworthy.”

In an alternate reality where nutrition science was functioning correctly, Bodnaruc et al.’s paper would be like a grenade thrown into the field. How disturbing, you might think, to discover that our North Star nutrition policy is possibly built on a foundation of sand! And, of course, there’s the added, bleak observation that Americans continue to get dramatically fatter and sicker while, on the whole, following the USDA's recommendations.

Yet most nutritionists refuse even to acknowledge criticisms like these. The guidelines are their gospel. They might object on the margins to, say, the lack of stricter advice on alcohol, sugar, and ultra-processed foods, or the need to consider environmental sustainability (links are mostly to Harvard nutritionists since they are by far the most influential in the U.S.). But the basic guidelines tenets--eat more fruits, vegetables, whole grains, nuts, seeds, legumes, and seafood along with limited amounts of lean meat and low-fat dairy—are nearly universally agreed upon. As summed up by the ubiquitously quoted New York University professor emeritus Marion Nestle, “Eat vegetables, fruit, and whole grains; don’t eat too many calories, red meat, salt, sugar, and saturated fat.” (or some such version).⁴

Never mentioned in these news stories is that Nestle herself helped write the guidelines in 1995. She served on an 11-member Dietary Guidelines Advisory Committee, as have many on the Harvard team—at least one Harvard representative has served on every advisory committee in recent memory. By any definition, this stands as a conflict of interest for them all since they are likely to be intellectually and professionally committed to preserving their work.

Regarding serious methodological issues, our primary concern here, these nutritionists have either ignored challengers or outright tried to silence them. Approximately zero nutritionists responded to the shocking findings in the NASEM report or our PNAS Nexus paper. And about 170 of them signed a letter urging the ultimately unsuccessful retraction of a BMJ cover story I wrote in 2015 challenging many aspects of the guidelines.

The Rebuttal