Search results

Randomized controlled trials (RCTs) are designed to answer causal questions with internal validity. However, threats to internal validity exist for even well-designed RCTs. In this article, we focus on how preregistration can help address some specific threats to internal validity related to the reporting of results. Preregistration involves researchers publicly posting critical decision points in a study prior to conducting it for the purpose of making researcher plans transparent, making deviations from those plans discoverable, and improving the validity of tests of significance. We provide a brief overview of null-hypothesis significance testing; consider how questionable research practices (e.g., p-hacking) and conducting data-dependent analysis threaten the validity of significance tests; discuss how preregistration can help address these threats and how preregistration works for RCTs; note limitations and challenges to preregistration; and provide recommendations for increasing the use of preregistration by researchers conducting RCTs in social work, education, and related fields.

Back in January, I posted about an article that was getting some attention in my world. Megan T. Stevenson is an active researcher in the criminal-justice-and-economics literature. She argues that when you look at the published studies that use randomized control trial methods to evaluate ways of reducing crime, most of the studies don't show a … Continue reading Pushback on Pessimism About Randomized Controlled Trials
The post Pushback on Pessimism About Randomized Controlled Trials first appeared on Conversable Economist.

Randomized controlled trials: the basics -- Types of randomized controlled trials -- Bias in randomized controlled trials -- Assessing the quality of randomized controlled trials: why, what, how, and by whom? -- Reporting and interpreting individual trials: the essentials -- From individual trials to groups of trials: reviews, meta-analyses and guidelines -- From trials to decisions: the basis of evidence-based health care -- Ethics and clinical trials: are they compatible? -- Reprise: more musings

Abstract
We report the results of a forecasting experiment about a randomized controlled trial that was conducted in the field. The experiment asks Ph.D. students, faculty, and policy practitioners to forecast (1) compliance rates for the RCT and (2) treatment effects of the intervention. The forecasting experiment randomizes the order of questions about compliance and treatment effects and the provision of information that a pilot experiment had been conducted which produced null results. Forecasters were excessively optimistic about treatment effects and unresponsive to item order as well as to information about a pilot. Those who declare themselves expert in the area relevant to the intervention are particularly resistant to new information that the treatment is ineffective. We interpret our results as suggesting that we should exercise caution when undertaking expert forecasting, since experts may have unrealistic expectations and may be inflexible in altering these even when provided new information.

In randomized controlled trials, treatment is often assigned by stratified randomization. I show that among all stratified randomization schemes that treat all units with probability one half, a certain matched-pair design achieves the maximum statistical precision for estimating the average treatment effect. In an important special case, the optimal design pairs units according to the baseline outcome. In a simulation study based on datasets from ten randomized controlled trials, this design lowers the standard error for the estimator of the average treatment effect by 10 percent on average, and by up to 34 percent, relative to the original designs. (JEL C13, C21)

Little research has deliberately investigated the effects of participant preferences for treatment condition in unblinded randomized controlled trials. We designed a study with a non-patient sample comparing a randomized arm to a preference arm of the same trial to investigate: (1) whether having a choice to select one's preference affects feelings about participation, belief in treatment effectiveness, treatment contamination, intervention adherence and engagement, and trial attrition; and (2) the interaction of preferences and treatment assignment on these variables. Contamination and attrition were rare and excluded from analyses. There was no effect of choice. Participants mismatched to preference felt less positive about their experience, but this did not affect belief in treatment, adherence, or engagement. Stronger effects may occur for patient populations.

ABSTRACTA pregnant woman was enrolled in a double‐blind randomized controlled trial (RCT) in which participants were randomized to a placebo or a drug being tested to prevent a hypertensive complication. After completing the trial, the research participant insisted on being told which drug she received to prepare for a future pregnancy. This case highlights an element of RCT procedure that has received minimal attention—whether to unblind study participants at the end of their participation. Given that unblinding is not standard practice for nonserious adverse events, what actions are justifiable for the investigators to take? To synthesize the information about this case, we used the CASES model, created by the National Center for Ethics in Health Care to analyze ethics cases. Ethical principles that guide research emphasize communication with participants and the importance of reducing harm within the constraints of the scientific goals. Participants may value knowing which drug they received for future health care decision‐making. We review information about the benefits and harms of unblinding.