METHODS: We searched Medline and the Cochrane Database of Systematic Reviews from 1 January 2018 to 31 December 2021 for systematic reviews and meta-analysis reports that include a trial sequential analysis. Only studies with at least two randomised clinical trials analysed in a forest plot and a trial sequential analysis were included. Two independent investigators assessed the studies. We evaluated protocolisation, reporting, and interpretation of the analyses, including their effect on any GRADE evaluation of imprecision.
RESULTS: We included 270 systematic reviews and 274 meta-analysis reports and extracted data from 624 trial sequential analyses. Only 134/270 (50%) systematic reviews planned the trial sequential analysis in the protocol. For analyses on dichotomous outcomes, the proportion of events in the control group was missing in 181/439 (41%), relative risk reduction in 105/439 (24%), alpha in 30/439 (7%), beta in 128/439 (29%), and heterogeneity in 232/439 (53%). For analyses on continuous outcomes, the minimally relevant difference was missing in 125/185 (68%), variance (or standard deviation) in 144/185 (78%), alpha in 23/185 (12%), beta in 63/185 (34%), and heterogeneity in 105/185 (57%). Graphical illustration of the trial sequential analysis was present in 93% of the analyses, however, the Z-curve was wrongly displayed in 135/624 (22%) and 227/624 (36%) did not include futility boundaries. The overall transparency of all 624 analyses was very poor in 236 (38%) and poor in 173 (28%).
CONCLUSIONS: The majority of trial sequential analyses are not transparent when preparing or presenting the required parameters, partly due to missing or poorly conducted protocols. This hampers interpretation, reproducibility, and validity.
STUDY REGISTRATION: PROSPERO CRD42021273811.
STUDY DESIGN AND SETTINGS: The Online Randomized Controlled Trials of Health Information Database was used as the sampling frame to identify a subset of self-recruited online trials of self-management interventions. The authors cataloged what these online trials were assessing, appraised study quality, extracted information on how trials were run, and assessed the potential for bias. We searched out how public and patient participation was integrated into online trial design and how this was reported. We recorded patterns of use for registration, reporting, settings, informed consent, public involvement, supplementary materials, and dissemination planning.
RESULTS: The sample included 41 online trials published from 2002 to 2015. The barriers to replicability and risk of bias in online trials included inadequate reporting of blinding in 28/41 (68%) studies; high attrition rates with incomplete or unreported data in 30/41 (73%) of trials; and 26/41 (63%) of studies were at high risk for selection bias as trial registrations were unreported. The methods for (23/41, 56%) trials contained insufficient information to replicate the trial, 19/41 did not report piloting the intervention. Only 2/41 studies were cross-platform compatible. Public involvement was most common for advisory roles (n = 9, 22%), and in the design, usability testing, and piloting of user materials (n = 9, 22%).
CONCLUSION: This study catalogs the state of online trials of self-management in the early 21st century and provides insights for online trials development as early as the protocol planning stage. Reporting of trials was generally poor and, in addition to recommending that authors report their trials in accordance with CONSORT guidelines, we make recommendations for researchers writing protocols, reporting on and evaluating online trials. The research highlights considerable room for improvement in trial registration, reporting of methods, data management plans, and public and patient involvement in self-recruited online trials of self-management interventions.
MAIN BODY: As an exemplar, we reflect on how, in the Asthma UK Centre for Applied Research (AUKCAR), we set out to create a supportive, organised environment with the overarching value of 'keeping patients at the heart of everything we do'. The key has been in planning and creating a suitably funded organisational infrastructure with dedicated PPI researchers along with the development of and expectation to abide by an agreed set of norms and values. Specifically, expecting AUKCAR PhD students and early career researchers to engage with PPI has established a working mode that we hope will last. Regular interactions and proactive Patient Leads increase PPI network cohesion.
CONCLUSION: With adaptation, the AUKCAR PPI model can be translated to international contexts.
METHODS: We reviewed measures of decision quality and decision process in 86 randomized controlled trials (RCTs) from the 2011 Cochrane Collaboration systematic review of PtDAs. Data on development of the measures, reliability, validity, responsiveness, precision, interpretability, feasibility, and acceptability were independently abstracted by 2 reviewers.
RESULTS: Information from 178 instances of use of measures was abstracted. Very few studies reported data on the performance of measures, with reliability (21%) and validity (16%) being the most common. Studies using new measures were less likely to include information about their psychometric performance. The review was limited to reporting of measures in studies included in the Cochrane review and did not consult prior publications.
CONCLUSIONS: Very little is reported about the development or performance of measures used to evaluate the effectiveness of PtDAs in published trials. Minimum reporting standards are proposed to enable authors to prepare study reports, editors and reviewers to evaluate submitted papers, and readers to appraise published studies.