![]() p. 381) asserted that experimenters “should feel entirely uninhibited about continuing or discontinuing” their experiments and even changing the stopping rule along the way, and corrections for sequential testing have more recently been described as “anathema” to Bayesian reasoning (Wagenmakers et al., 2018). In contrast, some authors favoring Bayesian analytic approaches (e.g., stopping testing once a critical Bayes Factor has been reached) argue that sequential testing is ‘no problem for Bayesians’ (Rouder, 2014), and that the stopping rule is ‘irrelevant’ from a Bayesian perspective (Edwards et al., 1963 Lindley, 1957 Wagenmakers, 2007). We aim to show that although unfettered use of sequential testing may raise problems, carefully designed procedures can limit the pitfalls arising from its use, allowing researchers to capitalize on the benefits it provides.Īs sequential testing involves repeated analyses, careful corrections must be made when using frequentist statistical approaches so as to avoid inflation of type 1 errors (see Lakens, 2014 for an accessible overview Wald, 1945). As a practical supplement to the issues we raise, we introduce an evolving resource aimed at helping researchers navigate both the statistical and psychological pitfalls of sequential testing: the Sequential Testing Hub (The site includes a guide for involving an independent analyst in a sequential testing pipeline, an annotated bibliography of relevant articles covering statistical aspects of sequential testing, links to tools and tutorials centered around how to actually implement a sequential analysis in practice, and space for suggestions to help develop this resource further. We discuss different ways of achieving this, from automation to collaborative inter-lab approaches. We argue for the consideration of an ‘insulated’ sequential testing approach, in which research personnel remain blind to the results of interim analyses. Without care, psychological factors may result in violations of this assumption when sequential testing is used: researchers’ behavior may be changed by the observation of incoming data, in turn influencing the process under investigation. ![]() An important but largely neglected assumption of sequential testing is that the data generating process under investigation remains constant across the experimental cycle. Although there are approaches that can mitigate many statistical issues with sequential testing, we suggest that current discussions of the topic are limited by focusing almost entirely on the mathematical underpinnings of analytic approaches. Sequential testing enables researchers to monitor and analyze data as it arrives, and decide whether or not to continue data collection depending on the results.
0 Comments
Leave a Reply. |