{"id":872,"date":"2019-06-24T16:00:45","date_gmt":"2019-06-24T16:00:45","guid":{"rendered":"http:\/\/replicationmarkets.com\/?p=872"},"modified":"2019-10-21T10:03:11","modified_gmt":"2019-10-21T14:03:11","slug":"what-is-high-powered-replication","status":"publish","type":"post","link":"https:\/\/replicationmarkets.org\/index.php\/2019\/06\/24\/what-is-high-powered-replication\/","title":{"rendered":"What is high quality replication?"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Note: details section (“Actually”) updated 21-OCT-2019. Additions shown in blue, deletions in strikethrough<\/del>.<\/em><\/span><\/p>

\u00a0<\/p>

In brief:<\/em> A good-faith, high-power attempt to reproduce a previously-observed finding.<\/p>

Slightly longer<\/em>: A good-faith attempt to reproduce a previously-observed finding, with a sample large enough to find the effect if there.<\/p>

We will presume you already know\u00a0what is a replication<\/a>. \u00a0So… what is \u201chigh quality\u201d? Ideal first, then actual.<\/p>

Ideally<\/h3>

An ideal<\/em> high-quality study would be definitive: if there is an effect, it will be found, and if not, it won’t. No single study can reach this ideal, but if we did a bunch of replications, all in different labs\u00a0we might come close. Multiple replications smooth out individual quirks or mistakes, and jointly they provide a large enough sample size that it has nearly 100% power to detect a real effect \u2013 even if it\u2019s notably smaller than the original paper claimed – as most are. That’s not practical for SCORE, but amazingly it has<\/em> been done at smaller scale — see for example\u00a0the Many Labs 2<\/a> study.\u00a0<\/p>

Such a study would come close to measuring what we\u00a0really<\/em> want to know: can we trust the claim? SCORE’s program description<\/a> says:<\/p>

Confidence scores are quantitative measures that should enable a … consumer of SBS [Social and Behavioral Science] research to understand the degree to which a particular claim or result is likely to be reproducible or replicable.\u00b9<\/p><\/blockquote>

This ideal definition abstracts away from the limits of any particular replication, which itself may get an unlucky draw. \u00a0Therefore, DARPA suggested SCORE forecasters consider this ideal scenario:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t \t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

If you’re now trying to specify sampling distributions for studies, stop. \u00a0This is the ideal. The point is to imagine sufficient effort that everyone would agree the result is a reliable Confidence Score.\u00a0<\/span>If all our forecasters kept this frame in mind, and forecast well, we would get the desired Confidence Scores. In fact, if we’ve idealized properly, it should be a very good estimate of how much they believe the claim is true<\/em>.<\/span>\u00a0<\/p>

But full disclosure: our accuracy isn’t based on the ideal, because ain\u2019t nobody got time to run 100 replications each on 150+ claims<\/em>. (At only $10K apiece, that would be $150M!) \u00a0<\/span>It’s based on one high-quality replication. \u00a0The forecast may be similar, but it’s unlikely to be the same. \u00a0<\/span><\/p>

So what are you really forecasting? What is “high quality” really?\u00a0<\/span><\/p>

Actually<\/b><\/h3>

SCORE’s TA1 team will run a single high-quality replication of each selected claim. \u00a0We assume good faith and competence, so \u201chigh quality\u201d amounts to \u201chigh power\u201d. The power must be high enough that a failed replication is informative about the truth of the claim. Simonsohn (2015<\/a>) argues for 2.5x the original sample size \u2013 a good rule but sometimes wasteful or practically unreachable.<\/p>

TA1 will use the two-stage approach from SSRP, noting cases where it is not feasible.<\/span><\/p>

The sampling strategy is similar to the one used in the original studies (students or other easily accessible adult subject pools) and are described in the Replication Reports for each replication posted at OSF (https:\/\/osf.io\/pfdyw\/<\/a><\/span>). For sample sizes we used a two-stage procedure with 90% power to detect 75% of the original effect size at the 5% level (two-sided test) in Stage 1; if the effect was not significant in the original direction in Stage 1 a second data-collection was carried out with 90% power to detect 50% of the original effect size at the 5% level (two-sided test) in the pooled first and second stage data collection.<\/span>\u00a0<\/span><\/p><\/blockquote>

Previous plan below, struck out.<\/span><\/p>

Therefore the TA1 gets three estimates of replication sample size, and aims for the middle:<\/del><\/p>