The metaumbrella package offers several facilities to assist in data analysis when performing an umbrella review. More precisely, this package is built around three core functions which aim to facilitate (i) the completion of the statistical analyses required for an umbrella review, (ii) the stratification of the evidence and (iii) the graphical presentation of the results of an umbrella review.
An associated app is also available to conduct umbrella reviews with a graphical user interface, directly on your web browser.
In this document, we present a very brief description of the core functions available in the metaumbrella package. Then, we give several concrete examples of umbrella reviews conducted in R using this package.
The package includes 3 core functions:
umbrella()
function
add.evidence()
function
forest()
function
1. umbrella() The umbrella()
function allows to
perform the calculations required to stratify the evidence. The main
argument required by this function is a well-formatted dataset. How
properly formatting your dataset is beyond the scope of this vignette
but general guidance can be found in the manual of this package and
another vignette is specifically dedicated to this issue.
Once your
dataset has been correctly formatted, it is used as an argument of the
umbrella function, which automatically:
For example, using df.SMD
as a well-formatted dataset,
you can automatically perform all the calculations required for an
umbrella review using this R code:
2. add.evidence() The add.evidence()
function
uses the calculations performed by the umbrella()
function
to perform a stratification of evidence according to algorithmic
criteria:
Ioannidis criteria
To obtain a stratification of evidence according to the Ioannidis
criteria, it only requires to specify
criteria = "Ioannidis"
in the add.evidence()
function.
GRADE criteria
To obtain a stratification of evidence according to the GRADE
criteria, it only requires to specify criteria = "GRADE"
in
the add.evidence()
function.
Personalized criteria
Up to 13 criteria can be used to stratify evidence in this Personalized classification.
In contrast to the two previous classifications, the Personalized
criteria requires to manually indicate several cut-off values for each
criteria you plan to use. Examples of stratification of the evidence
according to the ‘Personalized’ classification can be found in the
Example 3 and Example 4 of this vignette.
A brief example of stratification of the evidence according to the Personalized classification can be:
umb <- umbrella(df.SMD)
strat.prso <- add.evidence(umb, criteria = "Personalized",
class_I = c(total_n = 600, I2 = 25, rob = 75),
class_II = c(total_n = 400, I2 = 50, rob = 50),
class_III = c(total_n = 200, I2 = 75, rob = 25),
class_IV = c(total_n = 100))
3. forest() The forest()
function allows to have
a visualization of the results of the umbrella review
This example uses the dataset named df.OR
distributed along
with the metaumbrella package. You can access and visualize the dataset
in R with the following command
Because the dataset includes four factors (ASD, ADHD, ID and dyslexia), the calculations and the stratification of evidence will be performed independently for each of these factors.
To perform the calculations, simply apply the umbrella function on this well-formatted dataset.
This output shows the results of the calculations conducted by the
umbrella()
function. Results are presented independently
for each factor included in the dataset.
Once calculations have been performed via the umbrella function, you
can stratify the evidence with the add.evidence()
function.
Here, we present an example of stratification according to the
“Ioannidis” criteria.
A visual description of the results can be obtained using the
forest()
function. More information on how generating nice
plots using the forest()
function can be found in another
vignette dedicated to this function.
This example uses the dataset named df.RR
distributed
along with the metaumbrella package. You can access and visualize the
dataset in R with the following command
The dataset includes only one factor. To perform the calculations
required for the stratification of evidence, simply apply the umbrella
function on the df.RR
well-formatted dataset.
This output shows the results of the calculations for the factor included in the dataset.
Once the calculations have been performed via the umbrella function,
you can stratify the evidence with the add.evidence()
function. Here, we present an example of stratification according to the
“GRADE” criteria.
A visual description of the results can be obtained using the
forest()
function. More information on how generating nice
plots using the forest()
function can be found in another
vignette dedicated to this function.
This example uses the dataset named df.SMD
distributed
along with the metaumbrella package. You can access and visualize the
dataset in R with the following command
Because the dataset includes two factors, the umbrella review will consider these factors as independent. The calculations and the stratification of evidence will be performed independently for these two factors.
To perform these calculations, simply apply the umbrella function on
the df.SMD
well-formatted dataset.
This output shows the results of the calculations for the two factors included in the dataset.
In this example, we stratify evidence according to
Personalized
criteria. We take into account the number of
cases, the excess significance bias and the proportion of participants
in studies at low risk of bias.
For the number of cases (n_cases), we set the following criteria:
Class I: requires N to be > 800
Class II: requires N to be <= 800 but > 500
Class III: requires N to be <= 500 but > 200
Class IV: requires N to be <= 200 but > 100
Class V: implicitly requires N to be <= 100
This translates into this R code
strat.pers1 <- add.evidence(umb.SMD, criteria = "Personalized",
class_I = c(n_cases = 800),
class_II = c(n_cases = 500),
class_III = c(n_cases = 200),
class_IV = c(n_cases = 100))
For the excess significance bias (esb_p), we set the following criteria:
Class I: requires the p-value of the esb test to be > .10
Class II: requires the p-value of the esb test to be <= .10 but > .05
Class III: requires the p-value of the esb test to be <= .05 but > .01
Class IV: implicitly requires the p-value of the esb test to be <= .01
Class V: with these cut-off scores, a class V cannot be assigned based on the p-value of the esb test (a p-value < .01 leads to a class IV at the lowest).
This translates into this R code
strat.pers1 <- add.evidence(umb.SMD, criteria = "Personalized",
class_I = c(n_cases = 800, esb_p = .10),
class_II = c(n_cases = 500, esb_p = .05),
class_III = c(n_cases = 200, esb_p = .01),
class_IV = c(n_cases = 100))
For the proportion of participants included in studies at low risk of bias (rob), we set the following criteria:
Class I: % of participants included in studies at low risk of bias > 80%
Class II: % of participants included in studies at low risk of bias <= 80% but > 65%
Class III: % of participants included in studies at low risk of bias <= 65% but > 50%
Class IV: % of participants included in studies at low risk of bias <= 50% but > 35%
Class V: implicitly requires a % of participants included in studies at low risk of bias <= 35%
This translates into this R code
You can obtain the stratification of evidence via the standard
summary
command
A visual description of the results can be obtained using the
forest()
function. More information on how generating nice
plots using the forest()
function can be found in another
vignette dedicated to this function.
This example uses the dataset named df.OR.multi
distributed along with the metaumbrella package. You can access and
visualize the dataset in R with the following command
The dataset describes an umbrella review of meta-analyses of RCTs assessing the efficacy of several nutritional interventions on binary outcomes.
To perform the calculations required to stratify evidence, simply
apply the umbrella function on the well-formatted dataset. Because
multiple studies have several effect sizes, you have to indicate to the
umbrella function that the data have a multilevel structure by
specifying the mult.level = TRUE
argument. Moreover, to
apply the Borenstein method for the multiple outcomes, the correlation
between outcomes had to be specified with the r
argument of
the umbrella function (by default, the umbrella function assumes an
unique r = 0.5) or the r
column of the dataset.
Here, we assume that the study of Godebu has a mean correlation between
outcomes of .30 while all other studies have a mean correlation between
outcomes of .60. The r
argument of the umbrella function
accepts only one value. To have varying within-study correlations across
multivariate studies, you have to use the r
column of the
dataset. If a multivariate study has no r
value in the
dataset, the correlation indicated in the r
argument of the
umbrella function is used.
df.OR.multi$r <- NA # we initialize the r column in the dataset
df.OR.multi[df.OR.multi$author == "Godebu", ]$r <- .30 # we indicate a mean correlation of .30 for the study of Godebu
# option 1: we specify - via the r argument of the umbrella function - that all studies with multiple outcomes
# but no r values in the dataset are assigned with a correlation of .60.
umb.OR.multi_1 <- umbrella(df.OR.multi, mult.level = TRUE, r = 0.6)
# option 2: we manually specify - via the r argument of the dataset - the correlation for other studies
df.OR.multi[df.OR.multi$multiple_es == "outcomes" &
!is.na(df.OR.multi$multiple_es) &
!df.OR.multi$author %in% c("Godebu"), ]$r <- .60
# you no longer have to specify the r value in the umbrella function as it is already specified for all studies in the dataset
umb.OR.multi_2 <- umbrella(df.OR.multi, mult.level = TRUE)
# as usual, you can obtain results of the calculations using the summary command
summary(umb.OR.multi_2)
# check: you can check results are equal regardless of the method used
all(summary(umb.OR.multi_1) == summary(umb.OR.multi_2), na.rm = TRUE)
## [1] "all(summary(umb.OR.multi_1) == summary(umb.OR.multi_2), na.rm = TRUE) returns TRUE"
Once the multivariate structure of the data has been indicated in the umbrella function, the stratification of evidence is performed as for regular data.
In this example, we stratify evidence according Personalized criteria. We take into account the inconsistency, the small-study effects, the statistical significance of the largest study and the imprecision.
For the inconsistency, we set the following criteria:
Class I: requires an I² value < 0.20
Class II: requires an I² value >= 0.20 but < 0.40
Class III: requires an I² value >= 0.40 but < 0.60
Class IV: requires an I² value >= 0.60 but < 80
Class V: implicitly requires an I² value >= 80
This translates into this R code
For the small-study effects, we set the following criteria:
Class I: requires a p-value at the egger test > .10
Class II: requires a p-value at the egger test > .10
Class III: requires a p-value at the egger test <= .10 but > .05
Class IV: requires a p-value at the egger test <= .10 but > .05
Class V: implicitly requires a p-value at the egger test <= .05
This translates into this R code
For the significance of the largest study, we set the following criteria:
Class I: requires that the largest study has a p-value < .05 (i.e., the 95% CI excludes the null value)
Class II: requires that the largest study has a p-value < .05 (i.e., the 95% CI excludes the null value)
Class III: requires that the largest study has a p-value < .05 (i.e., the 95% CI excludes the null value)
Class IV: can be assigned if the p-value of the largest study is >= .05 (i.e., the 95% CI includes the null value)
Class V: with these cut-off scores, a class V cannot be assigned based on the p-value of the largest study (if the 95% CI includes the null, a Class IV can be assigned at the lowest based on this criteria).
This translates into this R code
For the imprecision, we set the following criteria:
Class I: requires that the meta-analysis has a power >= 80% to detect a SMD of 0.2
Class II: requires that the meta-analysis has a power < 80% to detect a SMD of 0.2 but a power >= 80% to detect a SMD of 0.4
Class III: requires that the meta-analysis has a power < 80% to detect a SMD of 0.4 but a power >= 80% to detect a SMD of 0.6
Class IV: requires that the meta-analysis has a power < 80% to detect a SMD of 0.6 but a power >= 80% to detect a SMD of 0.8
Class V: implicitly requires that the meta-analysis has a power < 80% to detect a SMD of 0.8
This translates into this R code
strat.pers2 <- add.evidence(umb.OR.multi_1, criteria = "Personalized",
class_I = c(I2 = 20, egger_p = .10, largest_CI = "notnull", imprecision = 0.2),
class_II = c(I2 = 40, egger_p = .10, largest_CI = "notnull", imprecision = 0.4),
class_III = c(I2 = 60, egger_p = .05, largest_CI = "notnull", imprecision = 0.6),
class_IV = c(I2 = 80, egger_p = .05, imprecision = 0.8))
Once these criteria have been indicated, you can obtain the
stratification of evidence via the standard summary
command
A visual description of the results can be obtained using the
forest()
function. More information on how generating nice
plots using the forest()
function can be found in another
vignette dedicated to this function.