Evaluating the Validity and Applicability of Systematic Reviews of Non-Randomized Studies
As a seasoned culinary professional, I’m well-versed in providing practical tips and in-depth insights on cooking, kitchen tools, knife skills, and culinary techniques. In this article, I’ll explore the key considerations for evaluating the validity and applicability of systematic reviews (SRs) of non-randomized dietary intervention studies.
Evidence in nutrition often comes from observational studies examining nutritional exposures, known as nutritional epidemiology studies. When using SRs of such studies to advise patients or populations on optimal dietary habits, it’s crucial for evidence users (e.g., healthcare professionals, policymakers) to first evaluate the rigor and utility of the SR.
Assessing the Validity and Applicability of Systematic Reviews
The first step in addressing the applicability of an SR of nutritional epidemiology studies is assessing whether the authors have stated explicit eligibility criteria that specify the population, exposure, comparator, and outcome(s) of interest (PECO). Unlike SRs of therapeutic interventions, where an intervention is compared to an alternative intervention or standard care, SRs of nutritional exposures typically compare individuals with higher intake to those with lower intake.
To be optimally useful, the SR should account for the foods or food compounds consumed instead of the exposure of interest, as these may also impact the risk for the outcome under study. Investigators may address this issue by summarizing results from substitution models or joint analyses, which estimate the effects of substituting one food or food compound for another or the joint effects of two or more exposures.
SRs are at risk of presenting misleading results if they fail to include all eligible studies. For most questions, SRs that search MEDLINE and EMBASE, or databases with similar coverage, likely include all or nearly all relevant published studies. However, for some questions, these databases may not be sufficient, and it’s difficult to know in advance whether a more extensive search is necessary.
When using an SR to inform inferences about the effect of a food or food compound on the risk for a health outcome, evidence users should look for results from a dose-response meta-analysis that summarizes the relationship between the quantity of the exposure and the risk for the outcome. Evidence users should also pay close attention to the quantity of the exposure corresponding to which results are presented and judge whether the quantity is reasonable.
Evaluating the Certainty of Evidence
Optimal decision-making also requires consideration of the certainty (quality) of evidence. The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach is the most commonly used system to evaluate the certainty of evidence.
The certainty of a body of evidence may be rated down by one or more levels due to concerns related to risk of bias, inconsistency, indirectness, imprecision, and publication bias. The certainty of a body of non-randomized studies may also be rated up in select scenarios, such as when there is a valid dose-response relationship or a large effect size.
Risk of bias in non-randomized studies may arise due to confounding, inappropriate selection of participants, errors in the measurement of the exposure, missing data, errors in the measurement of the outcome, and selective reporting of results. Evidence users should ensure that primary studies control for age, sex, smoking, and socioeconomic status, at a minimum.
When results across primary studies in an SR are inconsistent, we are less certain of the findings. Evidence users can look for inconsistency by visually inspecting a forest plot or by looking for statistical indicators of heterogeneity, such as the I^2 statistic.
Imprecision refers to the width of the confidence interval around an estimate. We are less certain of the pooled estimate if the lower and upper boundaries of the CI would lead to different dietary advice or actions.
Situations that may make us more certain of findings from non-randomized studies, such as large effect sizes or valid dose-response gradients, seldom occur in nutrition. As a result, SRs of nutritional epidemiology studies usually provide only low to very low certainty evidence.
Applying the Evidence to a Clinical Scenario
Let’s return to the opening clinical scenario of a 62-year-old Hispanic man with cardiovascular risk factors who is interested in switching to a Mediterranean-style diet.
The PREDIMED trial, a landmark RCT assessing the Mediterranean diet for major cardiovascular outcomes, is relevant to this scenario. While the trial faced some issues with subverted randomization, the investigators conducted adjusted analyses that provided reassurance about the validity of the results.
The trial demonstrated low to very low certainty evidence that the Mediterranean diet may result in small reductions in adverse cardiovascular and cancer health outcomes. Given the uncertainty and small magnitude of any potential benefit, the patient considers the inconvenience and reduction in the pleasure of eating not worth the possible benefits and chooses to continue her current levels of red and processed meat consumption from local, regenerative, and ethical sources.
In conclusion, evaluating the validity and applicability of SRs of non-randomized dietary intervention studies requires carefully considering the rigor of the review, the certainty of the evidence, and the relevance to the clinical or public health question of interest. By applying a structured approach, evidence users can make informed decisions to guide their patients or target populations towards optimal dietary habits.
Evaluating the Risk of Bias in Randomized Controlled Trials of Dietary Interventions
As a seasoned culinary professional, I’m well-versed in providing practical tips and in-depth insights on cooking, kitchen tools, knife skills, and culinary techniques. In this article, I’ll explore a structured approach for evaluating the risk of bias in randomized controlled trials (RCTs) of dietary interventions.
Bias is defined as a systematic deviation from the underlying truth due to a feature of the design or conduct of a research study. When evaluating the risk of bias in an RCT, there are several key considerations:
-
Prognostic Balance at Baseline: Successful randomization ensures that the intervention and control groups start with the same prognosis. This is achieved through concealment of the randomization allocation, which prevents those responsible for enrolling participants from manipulating the assignment.
-
Maintaining Prognostic Balance: Blinding of participants, healthcare providers, data collectors, outcome adjudicators, and data analysts helps maintain prognostic balance between study arms as the trial progresses.
-
Complete Follow-up: Knowing the status of each participant with respect to the outcomes of interest at the conclusion of the trial is crucial. High levels of loss to follow-up, particularly when missing data is not a random subset of all observations, can seriously undermine the credibility of study results.
-
Intention-to-Treat Analysis: Analyzing participants in the groups to which they were initially randomized, known as intention-to-treat analysis, offers a reliable method to maintain prognostic balance. Per-protocol and as-treated analyses can undermine the prognostic balance provided by randomization.
-
Early Stopping: Trials that are stopped too early, before enrolling the planned sample size, are at risk of substantially overestimating treatment effects, particularly when the number of participants and events is small.
Let’s apply this structured approach to evaluate the risk of bias in the PREDIMED trial, a landmark RCT assessing the Mediterranean diet for major cardiovascular outcomes.
The PREDIMED trial randomized 7,447 participants and experienced a 7% loss to follow-up, which is relatively low. The investigators followed the intention-to-treat principle and reported that the Mediterranean diet supplemented with nuts or extra-virgin olive oil reduced the incidence of major cardiovascular events.
However, the 2013 report demonstrated small between-group differences in some baseline characteristics, and a subsequent report in 2018 acknowledged that randomization appeared to have been subverted for 1,588 of the 7,447 participants. The investigators conducted adjusted analyses that provided reassurance about the validity of the results, but the trial was definitely at high risk of bias for being stopped early.
In summary, evaluating the risk of bias in RCTs of dietary interventions involves assessing prognostic balance at baseline, maintaining prognostic balance as the trial progresses, complete follow-up, intention-to-treat analysis, and early stopping. By applying a structured approach, evidence users can make informed judgments about the credibility of RCT results and their relevance to clinical or public health decision-making.
Interpreting the Magnitude of Effects and Applicability of Randomized Controlled Trials of Dietary Interventions
As a seasoned culinary professional, I’m well-versed in providing practical tips and in-depth insights on cooking, kitchen tools, knife skills, and culinary techniques. In this article, I’ll explore how to interpret the magnitude of effects and the applicability of randomized controlled trials (RCTs) of dietary interventions.
Interpreting the Magnitude of Effects
After evaluating the risk of bias in an RCT, the next step is to consider the magnitude of the observed effects and their precision. When reporting results, authors should present absolute effects, such as risk differences or number needed to treat or harm, rather than relying solely on relative effects (e.g., relative risk, odds ratio, hazard ratio).
Absolute effects provide a more realistic and intuitive understanding of the intervention’s impact, allowing evidence users to make rational decisions. Unfortunately, SRs of nutritional epidemiology studies often only present relative effects, as they tend to be similar across populations and provide more compelling (exaggerated) results.
When absolute effects are not presented, evidence users can calculate them using population risk estimates. For example, the NutriRECS systematic reviews of cohort studies presented absolute effect estimates corresponding to a reduction of red and processed meat by three servings per week, a reduction the authors thought might be feasible for most people.
Assessing the Applicability of the Evidence
Even when the results of an RCT appear highly effective, our certainty in the body of evidence may be undermined by issues such as limitations in study designs or differences between the questions addressed in studies and the clinical or public health question of interest. Hence, optimal decision-making also requires consideration of the certainty (quality) of evidence.
The Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach is the most commonly used system to evaluate the certainty of evidence. A body of evidence comprised of non-randomized studies, such as nutritional epidemiology studies, typically starts at low certainty due to concerns about residual confounding.
The certainty of a body of evidence may be rated down by one or more levels due to concerns related to risk of bias, inconsistency, indirectness, imprecision, and publication bias. The certainty of a body of non-randomized studies may also be rated up in select scenarios, such as when there is a valid dose-response relationship or a large effect size.
Let’s return to the opening clinical scenario of a 62-year-old Hispanic man with cardiovascular risk factors who is interested in switching to a Mediterranean-style diet. The PREDIMED trial demonstrated low to very low certainty evidence that the Mediterranean diet may result in small reductions in adverse cardiovascular and cancer health outcomes.
Given the uncertainty and small magnitude of any potential benefit, the patient considers the inconvenience and reduction in the pleasure of eating not worth the possible benefits and chooses to continue her current levels of red and processed meat consumption from local, regenerative, and ethical sources.
In conclusion, interpreting the magnitude of effects and the applicability of RCTs of dietary interventions involves considering both the absolute effects and the certainty of the evidence. By applying a structured approach, evidence users can make informed decisions to guide their patients or target populations towards optimal dietary habits.
Conclusion
As a seasoned culinary professional, I’ve explored the key considerations for evaluating the validity and applicability of systematic reviews (SRs) of non-randomized dietary intervention studies, as well as the risk of bias, magnitude of effects, and applicability of randomized controlled trials (RCTs) of dietary interventions.
When using SRs of observational studies to advise on optimal dietary habits, evidence users must first evaluate the rigor and utility of the review. This involves assessing whether the authors have stated explicit eligibility criteria, whether the review included an exhaustive search for eligible studies, and whether the review presented results in a useful manner.
Evaluating the certainty of evidence is also crucial, as SRs of nutritional epidemiology studies typically provide only low to very low certainty evidence due to concerns about risk of bias, inconsistency, indirectness, imprecision, and publication bias.
When interpreting RCTs of dietary interventions, evidence users should assess prognostic balance at baseline, maintenance of prognostic balance as the trial progresses, complete follow-up, intention-to-treat analysis, and early stopping. They should also consider the magnitude of the observed effects, preferably in terms of absolute effects, and the applicability of the evidence based on the certainty of the findings.
By applying a structured approach to evaluating the validity, applicability, and certainty of the evidence, culinary professionals and other evidence users can make informed decisions to guide their patients or target populations towards optimal dietary habits. This knowledge is essential for providing practical, evidence-based guidance on cooking, kitchen tools, knife skills, and culinary techniques.
https://pmc.ncbi.nlm.nih.gov/articles/PMC6852183/
https://nutrition.bmj.com/content/early/2024/07/24/bmjnph-2023-000833
https://pmc.ncbi.nlm.nih.gov/articles/PMC7307435/
