National Academies Press: OpenBook

Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief (2024)

Chapter: Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief

Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×
images Proceedings of a Workshop—in Brief

Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence

Proceedings of a Workshop—in Brief


In September and October 2023, the National Academies of Sciences, Engineering, and Medicine (the National Academies) Food and Nutrition Board held a virtual workshop series to discuss best practices for conducting meta-analyses (MAs) in nutrition research and utilizing MAs to inform policy. The three-part series, entitled Use of Meta-Analyses in Nutrition Research and Policy: A Workshop Series, explored the evidence on methods for planning, conducting, interpreting, and integrating the results of MA for use in nutrition research, policy development, and regulatory decision making. The workshop series was sponsored by the U.S. Food and Drug Administration (FDA) and featured invited presentations and discussions with researchers, government officials, and other global leaders in nutrition research and policy.

The first workshop in the series, held on September 19, 2023, focused on best practices for planning systematic reviews (SRs) and MAs for nutrition research. This initial workshop included opening remarks from two speakers from FDA about the critical importance of MAs to policy making at FDA and the agency’s need for a set of best practices to inform the use of MAs in nutrition policy development. Presenters and discussants addressed the foundational aspects of planning and delivering high-quality nutrition SRs and MAs.1 The second workshop in the series was held on September 25, 2023, to discuss the best practices for conducting SRs and MAs in the context of nutrition. Presenters discussed data errors, publication bias, and heterogeneity. The third and final workshop in the series which featured presentations on best practices in interpretation and application of SRs and MAs to evaluate the totality of evidence was held on October 3, 2023. This workshop addressed the impact of bias and data errors on interpreting results and the process for evaluating the strength of the evidence. As stated, the objective of the workshop series was to explore issues and best practices around MA; however, because MAs and SRs are closely related processes, many presentations included broader discussions around issues related to both MAs and SRs.

This Proceedings of a Workshop—in Brief2 will present highlights from the third workshop. This workshop was moderated by planning committee member Chizuru Nishida of the World Health Organization (retired) and featured presentations from Karima Benkhedda of Health Canada and

__________________

1 For discussion on the relationship between systematic reviews and meta-analyses, please see the Proceedings—in Brief for the first workshop in this series at https://nap.edu/27466.

2 Individual Proceedings of a Workshop—in Brief were prepared for each workshop in the series and are available online at https://nap.edu/27466 and https://nap.edu/27467. The full workshop proceedings is available at https://nap.edu/27481.

Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×

Barbara Schneeman of the University of California, Davis. The presentations were followed by a panel discussion with the presenters and additional discussants Elie Akl of the American University of Beirut and Vasanti Malik of the University of Toronto. The workshop series closed with concluding remarks from planning committee chair Katherine Tucker of the University of Massachusetts–Lowell.

The objectives of the third and final workshop were to

  • recognize the impact of risk of bias and publication bias on interpretation of results;
  • describe the impact of data errors on the conclusions of the SR and MA;
  • describe the process for evaluating the strength of the totality of evidence, with consideration of the type of study designs (observational and interventions) and associated risk of bias; and
  • describe the different applications of SRs and MAs to research and policy and considerations of evidence evaluation for each application.

The workshop presentations addressed the following questions, which were posed in advance by the workshop sponsor:

  • How to consider statistical heterogeneity when evaluating diet and disease relationships? Are higher levels of unexplained statistical heterogeneity acceptable for the field of nutrition? What are the best practices for addressing publication bias?
  • How to consider risk of bias when evaluating diet and disease relationships?
  • How can MA be used to evaluate the strength of the totality of evidence when there is evidence from different types of nutrition study designs?
  • How can MA be used to evaluate the strength of evidence when different outcomes are reported in different studies?

This Proceedings of a Workshop—in Brief highlights the presentations and discussions that occurred at the third workshop in the series and is not intended to provide a comprehensive summary of the information shared during the workshop.3 Statements, recommendations, and opinions expressed are those of individual presenters and participants and are not necessarily endorsed or verified by the National Academies of Sciences, Engineering, and Medicine; they should not be construed as reflecting any group consensus.

FROM SCIENCE TO POLICY: EVALUATING NUTRITION EVIDENCE FOR INFORMED DECISION MAKING

Benkhedda’s presentation focused on evaluating nutrition evidence for informed decision making and covered the use of SRs and MAs for the substantiation of health claims, using Health Canada’s policy and guideline development as a case study to illustrate the process. Benkhedda also addressed publication bias, heterogeneity, and risk of bias evaluation and their relevance for evaluating the totality of evidence. Benkhedda disclosed that she is part of the NuQuest working group, which develops risk of bias assessment tools for nutrition research, and that her presentation was developed using both published research, which is cited, and unpublished data.

Benkhedda defined a “health claim” as any presentation in labeling or advertising that states, suggests, or otherwise implies that a relationship exists between the consumption of a food or ingredient in the food and a person’s health. Exploring Health Canada’s approach to evaluating evidence to support health claims, Benkhedda spoke about two major guidance documents about how the industry can use literature to inform its health claims.4 Benkhedda said that while SRs are required to substantiate health claims in Canada, MAs are not. However, MAs can be useful for informing the review and evaluation of evidence.

Benkhedda detailed the systematic approach required for health claim substantiation (Figure 1). The process begins

__________________

Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×
Image
FIGURE 1 A systematic approach to health claims substantiation.
SOURCE: Presented by Karima Benkhedda on October 3, 2023 at the workshop on Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of the Evidence. ©All rights reserved. Guidance Document for Preparing a Submission for Food Health Claims, 2009. Health Canada. Reproduced with the permission of Health Canada, 2023. Available at https://www.canada.ca/en/health-canada/services/food-nutrition/legislation-guidelines/guidance-documents/guidance-document-preparing-submission-food-health-claims-2009-1.html (accessed January 10, 2024).

with a research question that forms the basis of the claim, then follows a set of steps culminating with a conclusion about the claim and developing the claim wording and the conditions under which the claim can be used.

Benkhedda explored the types of study designs that are used for the substantiation of health claims. She said that SRs, MAs, and randomized controlled trials (RCTs) are considered the highest level of evidence for health claims because they establish causality and provide information on intake-response relationships. She noted that prospective observational studies (i.e., cohort studies and nested case-control studies) could also be included but would be considered a lower quality of evidence because they show only association, have more confounders, are more prone to bias, and cannot establish causality. On the topic of bias, Benkhedda added that publication bias may impact a review. Although this concern is not unique to MAs, she noted that publication bias can be a major challenge with MAs due to their tendency to report on studies that show a significant effect. Benkhedda cautioned that having relevant studies missing from a review could adversely impact the intended decision making.

Case Study: Examining Evidence for an Impact of Diet on Cholesterol

Benkhedda detailed a real-world example of an MA that was performed to examine the impact of dietary changes on low-density lipoprotein (LDL) cholesterol. The MA included RCTs but contained high levels of statistical heterogeneity, and the team was concerned that missing studies could impact the results of the MA. In order to assess the absence of such studies, Benkhedda said that the team used funnel plots to visually assess the data. She noted that this method is imperfect and may perform poorly in settings with high heterogeneity because asymmetry in the funnel plot, which represents missing studies, can also occur as a result of the high heterogeneity. Benkhedda said it was important that they utilized additional methods of analysis to assess the quality of evidence, especially because the review would be used to inform decision making in a policy setting. She said that the team in this example also used subgroup analyses to explore the heterogeneity present in their selected studies, examining the study duration, participant demographics, and other relevant factors. They also tried removing studies with the strongest effects to assess the impact of their absence on the overall results of the MA. The bottom line, Benkhedda said, is that interpreting publication bias involves a combination of visual inspection of funnel plots, statistical tests, sensitivity analysis, and expert judgment.

Risk of Bias

Benkhedda described various domains of bias that may exist in nutrition studies. She highlighted that most risk of bias tools are not specifically designed to address the risks of bias that are most common in nutrition studies, but some tools have been adapted to address these issues. She described two quality appraisal tools used by Health Canada, including a quality appraisal tool for prospective observational studies. The tool asks questions such as whether the exposure was assessed more than once and whether the methodology was used to measure the exposure reported. She noted that confounders are also an area of concern in observational studies, and a question in the quality appraisal tool asks if confounders were corrected for in the study design or analysis process.

Benkhedda described an additional example from Health Canada, in which the evidence for a health claim on whole grains and coronary heart disease (CHD) was evaluated. The objective of the study was to determine whether the given evidence supported a health claim about whole grain foods and a reduced risk of CHD in the general population. The evaluation team used the Population, Intervention, Comparator, Outcome (PICO) framework to set up their SR (Box 1).

Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×

An SR of 26 RCTs and 6 cohort studies was conducted, and an MA of RCTs was performed for LDL and total cholesterol outcomes (Health Canada, 2012).5 Evidence from the RCTs showed a statistically significant effect, but when analyzing only the higher quality evidence or the evidence on grains other than those high in betaglucan fiber, this effect was lost. The data from cohort studies was too heterogeneous to be pooled, and most of the studies were deemed to be too low quality, with high risk of bias due to lack of adjustment for confounders and lack of control for potential confounders. In the end, the evidence was not sufficient to support a health claim about whole grains and CHD risk reduction.

Benkhedda addressed the question of how to consider risk of bias when evaluating diet and disease relationships. She stated the importance of considering the overall quality of evidence and referenced the previous example of the MA on whole grain intake and CHD, in which the evidence was not considered high enough quality to substantiate a claim. Referring to Figure 1, which depicted Health Canada’s approach to health claims substantiation, Benkhedda spoke about ways to assess causality, including considering the overall consistency of the evidence, the strength of association, and the intake-response (or dose-response) relationships. She explained that MAs can help to determine health claim validity by assessing these factors and the overall quality of evidence across studies. MAs can answer questions such as whether consistency was high across the higher-quality studies and if appropriate tests were used to quantify heterogeneity. Benkhedda suggested analyzing the proportion of studies that showed statistically significant effects and exploring their quality and what factors might have impacted the statistical significance of the non-significant studies. Benkhedda said that when considering dose-response relationships, the minimum effective amount shown to produce a response should be identified, and in observational studies, statistically significant differences found between the highest-intake groups and lowest-intake groups should be examined.

Evaluating Strength and Quality of Evidence

Benkhedda emphasized the importance of establishing the applicability of the MA results to the target population. For health claims, the target population is the general population of the country. Conducting subgroup analyses within the MA can help to establish the robustness of the effect for important subgroups of interest, with a focus on the physiological meaningfulness of a food’s effect. Benkhedda highlighted

__________________

Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×

the importance of quantifying the impact of the food’s exposure on human health.

Addressing a question of how MAs can be used to evaluate the strength of the totality of evidence when the evidence comes from different types of nutrition study designs, Benkhedda reiterated the benefits of a broad, holistic assessment of evidence. Consider the comprehensiveness, relevant outcomes, consistency and strength of association, quality, meaningfulness of the effect size, and the generalizability of the data, she suggested. Benkhedda also addressed the question of how MAs can be used to evaluate the strength of the evidence when different outcomes are reported in different studies. Benkhedda said that claim wording should reflect the specific evidence, and the claim should reflect whether the evidence showed prevention of disease, change in disease outcome, or change in disease risk biomarkers. She illustrated this point with another real-world example, in which Health Canada examined the evidence to support a health claim for fruit and vegetable consumption to reduce the risk of heart disease, and for evidence to support a health claim of whole grains consumption to reduce CHD risk. They found sufficient evidence to support the health claims for fruits and vegetables but did not find sufficient evidence to support the health claims for whole grains, because a sensitivity analysis revealed that the evidence for the whole grains health claim was in limited trials of grains high in betaglucan and in studies judged to be of poor quality, which were credited with producing most of the effect.

Considerations for the Use of Meta-Analyses in Nutrition Policy Development

Benkhedda concluded her presentation with an overview of the considerations for use of MA in policy development. She suggested using the best evidence available, including SRs, MAs, and relevant individual studies. Considerations should be given to the relevance of a study to the specific policy question, the overall quality of evidence, the level of certainty, and the applicability of the evidence to the general population in a national context. The advantage of MAs and SRs, she noted, is that they examine a large sample of studies under similar conditions and can draw conclusions that are relevant to policy development. However, Benkhedda noted that the included research should be relevant to the policy question, and she suggested that it may be helpful to conduct SRs in collaboration with policy makers.

NUTRITION AND POLICY: EVALUATING THE EVIDENCE

Schneeman’s presentation provided examples of the application of SRs and MAs in nutrition policy. She featured examples across three bodies that use nutrition research to develop guidelines and policies: FDA, the U.S. Dietary Guidelines Advisory Committee (DGAC), and the World Health Organization (WHO) Nutrition Guidance Expert Advisory Group.

The examples for discussion were the FDA’s use of SR evidence in acceptance, denial, or qualification of health claims; the DGAC’s use of SR evidence to produce a scientific report to advise the relevant federal agencies on updates to the Dietary Guidelines for Americans (DGA); and the WHO’s use of SRs and MAs in developing nutrition guidelines. Schneeman disclosed that she holds, or has held, affiliations with numerous stakeholder organizations, including the DGAC, WHO Nutrition Guidance Expert Advisory Group, the National Academies’ Food and Nutrition Board, and the International Union of Food Science and Technology Task Force on Food Classification. She also noted her position as an advisory board member or member of the board of trustees for organizations including the International Food Information Council, McCormick Science Institute, and the Coalition for Grain Fiber Science Advisory Committee.

Schneeman provided the WHO’s definitions of SRs and MAs. As she quoted, “a[n] SR is a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to extract and analyze data from the studies that are included in the review” (WHO, 2014). An SR is different from an MA, which refers to the quantitative synthesis, or pooling, of outcome data across comparable studies to achieve a pooled estimate of effect. Schneeman explained that if data from an SR meets certain requirements, such as high homogeneity across study design, population, and intervention,

Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×

then the data can be combined into an MA. An MA is a statistical method that provides a summary estimate of effect across a body of evidence.

Schneeman described the process of undertaking an SR across each of the three bodies. At FDA, the SR process is initiated in response to a petition, the agency’s interest in updating existing claims, or to determine whether there are any major and relevant omissions in the literature. The DGAC uses SRs to review the evidence on topics for inclusion in the next iteration of the DGA and U.S. Department of Agriculture Nutrition Evidence Systematic Review (NESR)6 has proposed using scoping reviews to identify potential topics for the DGA. At WHO, SRs are used to identify the availability of relevant evidence for guideline development and to facilitate protocol development.

Schneeman noted that it is important for policy makers to understand the overall quality of evidence and the extent to which the data has significant scientific agreement, and explained the importance of deciding what evidence to include and exclude from an SR. She emphasized the importance of having protocols in place to help ensure that all decision-making criteria are consistently applied. Specific, meaningful outcomes are important when evaluating evidence for health claims, Schneeman said. She said that it is critical to consider the intended use of evidence and what studies are and are not relevant to the research question.

To this end, Schneeman described how an MA can provide insight into the inconsistencies in the available evidence. She noted that MAs can be used to examine the differences between studies and emphasized the usefulness of subgroup analyses when subgroups may be relevant to the specific guidelines being developed. She noted that the DGAC and NESR team provide an analytical framework for understanding the key factors that may impact the relationships in nutrition studies, such as confounders, covariates, and moderators.

Structuring the Systematic Review

Schneeman described the processes of structuring an SR at the three entities of focus. She stated that research teams identify the population of interest, including characteristics such as age, sex, and health status, and the intervention or exposure, aiming for specificity to allow for identification of studies that are relevant to the policy question. She noted that FDA would be more interested in the impact of a substance than in the lack of that substance as the basis for the health claim.

At FDA, when an SR is developed in response to a health claim petition or as evidence for food labeling, the petition will be used to create the PICO elements, and the inclusion and exclusion criteria will be specified in the FDA guidance document. For the DGAC, NESR methodology specifies the inclusion and exclusion criteria that will be used in the SR, which must be relevant to the DGA and modified as needed to address specific questions. The SR is conducted by methodological experts. When an SR is performed for the WHO Nutrition Guidance Expert Advisory Group, a subcommittee determines the PICO elements, defines the critical outcomes of interest, and specifies the inclusion and exclusion criteria, using an approach that is consistent with the WHO handbook. Schneeman reinforced the importance of bringing methodology and subject matter experts into every step of the process.

Assessing Strength of Evidence

Schneeman discussed methods for assessing the strength of the evidence, and how these processes differ across the three bodies. At FDA, the focus is on whether the evidence is consistent with significant scientific agreement. The DGAC assigns each SR a grade, with criteria for risk of bias, consistency, precision, directness, and generalizability. WHO uses the Grading of Recommendations Assessment, Development and Evaluation (GRADE) methodology for rating evidence, which takes into account study design and allows researchers to evaluate and grade the quality of the evidence on a rating scale, from very high-quality evidence to very low-quality evidence. This approach, Schneeman said, may help to improve objectivity when assessing the strength of the evidence.

Use of Meta-Analyses in Decision Making

After grading and analyzing the evidence, Schneeman explained, the next phase is the decision-making

__________________

6 For more information on NESR, please see https://nesr.usda.gov/ (accessed January 16, 2024).

Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×

process. She explained that this phase of the process is where the three groups differ most in their approach. At FDA, the criteria and conditions related to the food products that might bear a claim must be considered as well as the legal and economic factors on the use of claims in labeling. The DGAC considers not only the evidence from SRs for its conclusions but also data analysis from the National Health and Nutrition Examination Survey to understand current intakes, as well as food pattern modeling to structure healthful dietary patterns. Schneeman explained that the DGAC integrates the evidence from these approaches to develop its conclusions. At WHO, SRs and MAs are integrated into decision making using a series of contextual factors to determine the strength of the recommendations. For example, WHO considers the balance of benefits and harms, resource implications, confidence in effect estimates, equity and human rights, and accessibility and feasibility.

Schneeman shared some observations from her experiences using MAs in the decision-making process. She stated that a well-designed SR is essential for a proper MA. She recommended involving methodology experts in designing the SR and performing the MA. She also recommended conducting a scoping review before an SR to set the SR’s parameters and focus on information critical to the decision making. Schneeman said that SRs should be designed to fit the purpose of the policy or guidance, and MAs should allow for a deeper examination of the evidence to better understand its strength. However, Schneeman said, when MAs are not possible, other tools such as harvest plots can be used.

Schneeman provided a brief overview of the opportunities and challenges for using MAs to develop guidance across the three bodies. She suggested that, at FDA, a graphic display (e.g., a forest plot) might better illustrate the balance of evidence and serve as a useful tool to clarify inconsistencies in the current evidence. For the DGAC, MAs could provide more transparency when rating the strength of the evidence. She also suggested that non-qualitative summary tools could facilitate recommendations related to the food environment and policy and that subgroup analyses could better help the committee understand what subgroups would benefit most from specific changes to the DGA. Schneeman noted that MAs are used for guideline development at WHO, which allows experts in methodology to assess the strength of the evidence and for expert guidance committees to use the MAs in the development of the recommendation. She pointed out that MAs could also be used to determine when evidence is insufficient to produce a recommendation.

PANEL DISCUSSION

Nishida led a panel discussion featuring Benkhedda and Schneeman, who were joined by Malik and Akl.

Tools to Assess the Quality of Evidence from Nutrition Research to Inform Policy

Planning committee member Russell de Souza, echoed by other audience members, asked whether the field of nutrition is so unique that it requires its own set of statistical and analytical tools. Akl replied, disclosing that his role as part of the GRADE working group has informed his perspective, and stated that while many fields may consider theirs to be unique, his opinion is that the principles should be the same across fields. The general principles across fields are that certainty in effect estimates is affected by risk of bias, inconsistency of results, imprecision, indirect evidence, and publication bias. However, Akl said, in certain fields, due to population size, small effect sizes may be more meaningful. What is judged as a small effect size in one field might be significant in another field.

Akl described some factors that he thinks are particularly relevant to the nutrition field. For example, in nutrition, it is common to have outcomes with continuous measurements. Continuous scales often show heterogeneity due to the nature of the data. This factor should be considered when assessing heterogeneity in nutrition research, Akl said, but not “over-rated,” which could lead to the downgrading of the certainty of evidence. He noted that FDA and DGAC use different approaches for different goals: validation of health claims and development of guidelines, respectively. Akl

Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×

emphasized the importance of involving content experts in the development and use of SRs and MAs.

Akl also encouraged the consideration of the contextual factors that impact nutrition policy and guideline development, such as the resources that would be required to implement or enforce a policy or recommendation and the acceptance of a recommendation by stakeholders, such as industry and consumers. He urged researchers to consider the real-world implications of a potential policy or guideline, suggesting that SRs that examine these contextual factors could be part of the methodologies that governing bodies use to inform their processes.

Malik suggested that the hierarchy of evidence pyramid (Figure 2) presented by Benkhedda may not be the ideal reference point for the field of nutrition (Yetley et al., 2017). Nishida concurred with this comment. Malik explained that the field of nutrition often examines longitudinal relationships between diet and health, which are best examined through cohort studies. Currently, RCTs are considered to be the highest level of evidence, with cohort studies viewed as lower quality, and Malik suggested that a restructuring of the pyramid be considered, with cohort and RCT side-by-side.

Benkhedda agreed that sometimes well-designed and well-conducted observational studies do yield better results than poor-quality RCTs. However, she noted that in validating health claims, Health Canada requires RCTs first, and then observational studies can provide additional support. If an observational study shows an effect, a follow-up RCT of the food or nutrient in question can help to clarify and determine whether a causal relationship truly exists. However, she agreed that for dietary guideline development, RCTs may not be the most important source of data, suggesting that further discussion on this topic among nutrition experts may be warranted.

Schneeman stated that the GRADE approach can be helpful because it allows for the upgrading or downgrading of certain studies relative to their efficacy in answering the research question. She also emphasized that different government bodies, and different countries, have specific legal and contextual factors to consider when developing nutrition policy. Schneeman added that while she used to think that nutrition studies were unique in their complexity and required unique tools, her thinking has evolved over her career, and she has come to understand that using well-defined tools consistently over time to facilitate improved understanding of the data can contribute to effective policy decision making as well as to transparency in the decision-making process. Benkhedda and Malik added their agreement with Schneeman and Akl that the consistent use of existing tools is key to their efficacy.

Image
FIGURE 2 A hierarchy of evidence for health claims substantiation used by Health Canada,
SOURCE: Presented by Karima Benkhedda on October 3, 2023 at the workshop on the Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence; Yetley et al., 2017. Reprinted with permission.
Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×

Conflict of Interest and Bias

The second major topic of conversation was the potential impacts of conflict of interest, including funding sources and other vested interests, on nutrition research results and ways to mitigate this bias. Benkhedda explained Health Canada’s approach for health claims validation, stating that Health Canada examines all evidence through a set of objective criteria, and that while they do not exclude evidence due to funding source, they consider whether the funding source may have biased the outcome of the study. For nutrition guidelines, Benkhedda noted that the committee developing the guideline may choose to exclude studies funded by industry.

Akl added that funding by industry or an interested party, or a conflict of interest from study authors, does not necessarily negatively impact a study; but it has to predict when they would and when they would not. Therefore, these represent a red flag that should be addressed because evidence shows that studies funded by industry have more favorable results than those that are not. He also explained that conflicts of interest can exist beyond funding, such as intellectual conflicts of interest among members of a committee or a reviewing body. If an expert comes to a panel with a preconceived notion about a recommendation, Akl explained, they may not have an open mind about the evidence being presented. This type of conflict of interest can be very challenging to detect, as it falls outside the standard conflict of interest declaration and management policies.

Schneeman added that for health claims and petitions to FDA, most of the evidence provided will be funded by industry. However, the evidence must still meet prespecified transparent criteria, which are created to ensure that the evidence is high quality. Malik added that it may be useful to conduct subgroup analyses by funding source. Also, the NutriGrade system has the option to screen data by funding source when using the tool to rank evidence in an MA.

CLOSING REMARKS

Tucker delivered closing remarks at the culmination of the workshop series, summarizing key takeaways from each of the three workshops. Tucker said that nutrition research is highly complex, and policy makers have to rely on systematic approaches to understanding the evidence, such as the results of SRs and MAs. Nutrition research is often used to inform policy and guidelines. Tucker emphasized the importance of systematic, consistent approaches in nutrition research studies.

Highlighting key learnings from the presentations by Benkhedda and Schneeman, Tucker spoke about Benkhedda’s description of Health Canada’s approach to developing guidance using evidence and the importance of using subgroup analysis to address heterogeneity. She also reinforced the benefits of appropriate and consistent use of high-quality appraisal and analysis tools. Tucker noted Schneeman’s discussion of critical frameworks, such as PICO and the NESR methodology, and how they are utilized across agencies to assess the strength of evidence for developing nutrition policies.

Tucker closed by expressing her appreciation for the discussions that took place across the three workshops in the series. She also thanked FDA, the National Academies, workshop speakers, attendees, and planning committee members for their integral roles in sponsoring, planning, organizing, and attending the workshop series.

REFERENCES

Health Canada. 2009. Guidance document for preparing a submission for food health claims. Health Canada. Bureau of Nutritional Sciences. https://www.canada.ca/en/health-canada/services/food-nutrition/legislation-guidelines/guidance-documents/guidance-document-preparing-submission-food-health-claims-2009-1.html (accessed February 25, 2024).

Health Canada. 2012. Summary of Health Canada’s assessment of a health claim about whole grains and coronary heart disease. Health Canada. Bureau of Nutritional Sciences. https://www.canada.ca/en/health-canada/services/food-nutrition/food-labelling/health-claims/assessments/assessment-health-claim-about-whole-grains-coronary-heart-disease.html (accessed December 31, 2023).

Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×

WHO (World Health Organization). 2014. WHO handbook for guideline development, 2nd ed. https://www.who.int/publications/i/item/9789241548960 (accessed December 31, 2023).

Yetley E. A., A. J. MacFarlane, L. S. Green-Finestone, C. Garza, J. D. Ard, S. A. Atkinson, D. M. Bier, A. L. Carriquiry, W. R. Harlan, D. Hattis, J. C. King, D. Krewski, D. L. O’Connor, R. L. Prentice, J. V. Rodricks, and G. A. Wells. 2017. Options for basing Dietary Reference Intakes (DRIs) on chronic disease endpoints: Report from a joint US/Canadian-sponsored working group. American Journal of Clinical Nutrition 105(1):249S–285S.

DISCLAIMER This Proceedings of a Workshop—in Brief has been prepared by Melissa Maitin-Shepard and Marian Flaxman as a factual summary of what occurred at the meeting. The statements made are those of the rapporteurs or individual workshop participants and do not necessarily represent the views of all workshop participants; the planning committee; or the National Academies of Sciences, Engineering, and Medicine.

*The National Academies of Sciences, Engineering, and Medicine’s planning committees are solely responsible for organizing the workshop, identifying topics, and choosing speakers. The responsibility for the published Proceedings of a Workshop—in Brief rests with the institution. The planning committee comprises Katherine Tucker (Chair), University of Massachusetts–Lowell; Mei Chung, Tufts University; Russell J. de Souza, McMaster University; Amanda MacFarlane, Texas A&M University; Chizuru Nishida, World Health Organization (retired); and Janet Tooze, Wake Forest University.

REVIEWERS To ensure that it meets institutional standards for quality and objectivity, this Proceedings of a Workshop—in Brief was reviewed by Russell J. de Souza, McMaster University and Vasanti S. Malik, University of Toronto. Leslie Sim, National Academies of Sciences, Engineering, and Medicine served as the review coordinator.

SPONSOR This workshop was supported by the United States Food and Drug Administration (contract no. 75F40122D00002/75F402122F19002).

STAFF Takyera Robinson, Alice Vorosmarti, and Samuel Crawford, Food and Nutrition Board, Health and Medicine Division, National Academies of Sciences, Engineering, and Medicine.

For additional information regarding the workshop, visit https://www.nationalacademies.org/event/40375_10-2023_use-of-meta-analyses-in-nutrition-research-and-policy-interpretation-and-application-of-meta-analysis-to-evaluate-the-totality-of-evidence.

SUGGESTED CITATION National Academies of Sciences, Engineering, and Medicine. 2024. Use of meta-analyses in nutrition research and policy: Interpretation and application of meta-analysis to evaluate the totality of evidence: Proceedings of a workshop—in brief. Washington, DC: The National Academies Press. https://doi.org/10.17226/27468.

Health and Medicine Division

Copyright 2024 by the National Academy of Sciences. All rights reserved.

images
Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×
Page 1
Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×
Page 2
Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×
Page 3
Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×
Page 4
Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×
Page 5
Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×
Page 6
Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×
Page 7
Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×
Page 8
Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×
Page 9
Suggested Citation:"Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop - in Brief." National Academies of Sciences, Engineering, and Medicine. 2024. Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief. Washington, DC: The National Academies Press. doi: 10.17226/27468.
×
Page 10
Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief Get This Book
×
 Use of Meta-Analyses in Nutrition Research and Policy: Interpretation and Application of Meta-Analysis to Evaluate the Totality of Evidence: Proceedings of a Workshop–in Brief
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The third workshop in the series on the use of meta-analysis in nutrition research and policy, held on October 3, 2023, focused on the process for evaluating the strength of the totality of evidence for diet and health relationships, with consideration of the type of study designs (observational and interventions) and risk of bias. The workshop series concluded with a discussion of the different applications of meta-analysis to inform policy and guidance.

READ FREE ONLINE

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!