Instruments to assess a study
What is the difference between an instrument assessing the methodological quality of a study, and an instrument assessing the quality of the reporting of a study? And what guidelines are available for randomised controlled trials and observational studies? The purpose of this article is to give the reader a quick overview in relation to these questions. It discusses methodological quality lists recommended by the Cochrane Collaboration, and the CONSORT, STROBE and PRISMA Statement for reporting of studies.
When I was teaching epidemiological study designs to MSc and MPH students at the University of Pretoria, they did a case study on the methodological quality of a Randomised Controlled Trial (RCT) using the Delphi list (1). I also discussed the CONSORT Statement 2010 and that led to questions on the difference between these lists.
In general there are two categories of instruments (which overlap to some degree): 1) instruments that assess the methodological quality of studies, and 2) instruments that assess the quality of the reporting. The Delphi list is an example of the first category, the CONSORT Statement of the second one.
To make it even more complex, there turn out to be numerous instruments (including modifications) that have been published. In the systematic reviews that are part of my PhD thesis I for example used a list developed by Tulder et. al (2).
And there are also instruments that e.g. assess the quality of guidelines, as discussed in the article in this newsletter on Assessing the quality of standard treatment guidelines.
The purpose of this article is to give the reader a quick overview of the different types of instruments, either to assess the methodological or reporting quality of studies, without being comprehensive.
Methodological quality of research
Reasons for performing quality assessment of studies include: to determine a minimum quality threshold for the selection of primary studies for a systematic review; to explore differences in quality as an explanation of heterogeneity in study results; to weigh the results in proportion to the quality in meta-analysis; and to guide interpretation of findings, determine the strength of inferences, and guide recommendations for future research and clinical practice.
RCTs
The Cochrane Collaboration uses a ‘Risk of bias’ (methodological quality) tool that review authors are expected to use for assessing risk of bias in RCTs. This involves consideration of six features: sequence generation, allocation sequence concealment, blinding, incomplete outcome data, selective outcome reporting and ‘other’ potential sources of bias. Items are assessed by: 1) providing a description of what happened in the study; 2) providing a judgment on the adequacy of the study with regard to the item. The judgment is formulated by answering a pre-specified question, such that an answer of ‘Yes’ indicates low risk of bias, an answer of ‘No’ indicates high risk of bias, and an answer of ‘Unclear’ indicates unclear or unknown risk of bias. This tool can also be used for experimental and controlled studies, and for prospective cohort studies, whether allocation is randomised or not.
Observational studies
The Cochrane Collaboration does not recommend a specific tool for assessing bias in observational studies. They indicate that many instruments for assessing methodological quality of non-RCTs of interventions have been created e.g. the Downs and Black instrument and the Newcastle-Ottawa Scale, and review authors should select (and modify) the instrument that is most applicable to their topic.
Quality of the reporting of studies
Incomplete and inadequate reporting of research hampers the assessment of the strengths and weaknesses of the studies reported in the medical literature. Readers need to know what was planned (and what was not), what was done, what was found, and what the results mean. Recommendations on the reporting of studies that are endorsed by leading medical journals can improve the quality of reporting.
CONSORT Statement
The CONSORT (CONsolidated Standards of Reporting Trials) Statement is intended to improve the reporting of a RCTs, enabling readers to understand a trial's design, conduct, analysis and interpretation, and to assess the validity of its results. The checklist items pertain to the content of the title, abstract, introduction, methods, results, discussion and other information. The checklist includes the 25 items selected because empirical evidence indicates that not reporting the information is associated with biased estimates of treatment effect, or because the information is essential to judge the reliability or relevance of the findings. The flow diagram is intended to depict the passage of participants through an RCT. The diagram explicitly shows the number of participants, for each intervention group, included in the primary data analysis. Inclusion of these numbers allows the reader to judge whether the authors have done an intention-to-treat analysis.
STROBE statement
Observational research comprises several study designs (e.g. cohort study, case-control study, cross-sectional study). The STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) Statement has a similar aim as the CONSORT Statement, but is designed for observational studies. The checklist covers 22 items under the headings title and abstract, introduction, methods, results, discussion and other information.
PRISMA statement
PRISMA stands for Preferred Reporting Items for Systematic Reviews and Meta-Analyses. It is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. The aim of the PRISMA Statement is to help authors improve the reporting of systematic reviews and meta-analyses. The focus is on RCTs, but PRISMA can also be used as a basis for reporting systematic reviews of other types of research, particularly evaluations of interventions. The PRISMA Statement consists of a 27-item checklist (containing sections similar to those of the other statements) and a four-phase flow diagram, focussing on identification, screening, eligibility and inclusion of studies in the review.
References:
- Verhagen AP et al. The Delphi list: a criteria list for quality assessment of randomised clinical trials for conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol 1998; 51:1235-41. Link to abstract
- van Tulder MW, Assendelft WJ, Koes BW, Bouter LM. Method guidelines for systematic reviews in the Cochrane Collaboration Back Review Group for Spinal Disorders. Spine 1997;22:2323-30.