{"id":951,"date":"2019-06-24T15:52:03","date_gmt":"2019-06-24T15:52:03","guid":{"rendered":"http:\/\/replicationmarkets.com\/?p=951"},"modified":"2020-02-25T08:51:11","modified_gmt":"2020-02-25T13:51:11","slug":"what-is-replication","status":"publish","type":"post","link":"https:\/\/replicationmarkets.org\/index.php\/2019\/06\/24\/what-is-replication\/","title":{"rendered":"What is replication?"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

[Updated Jan. 2020 to cover unified definition, and clarify]<\/span><\/p>\n

Concept<\/h4>\n

Replication is when you repeat a previous study to see if you get the same results. Ideally this happens a lot in science. In practice, not as much. But what counts?\u00a0<\/p>\n

According to Brian Nosek (“What is replication?”, 2019), people commonly say replication is repeating a study’s procedure<\/i> and observing whether the previous finding recurs, but this definition fails because the changes<\/i> to procedure define the replication.<\/p>\n

Consider: when replicating an Israeli study in the US, he didn’t use the original materials – they were in Hebrew!\u00a0Replications change participants, campus, country, language, etc. But if any of these negated something like the\u00a0Stroop effect, that would be a surprising and important discovery.\u00a0<\/p>\n

So, Nosek argues replication is really a conceptual notion.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t \t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

<\/p>\n

(He says \u201cno a priori<\/em>\u00a0reason\u201d — reasons made up\u00a0afterwards don’t count.\u00a0<\/em>\u00a0We also assume competent, good-faith replication.)<\/p>\n

Claiming something is a replication \u201cis a theoretical commitment.\u201d<\/p>\n

<\/p>\n

<\/p>\n

<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Practical Definition for DARPA SCORE<\/h4>\n

Originally, we distinguished between direct replications and data replications, and privileged direct replications.\u00a0Direct replications <\/b>test the original claim by gathering new data<\/em>, such as a new psychology experiment.<\/p>\n

Data-analytic replications<\/strong> test the original claim using new found<\/em> data, data appropriate for replication but not originally collected by the replication research team. For example, using the same economic indicator as the original-study researchers, but from a different time period.<\/p>\n

But direct replications exclude most economics, sociology, and political science. So, as of February 2020 (Round 6), we will adopt DARPA’s new unified definition:<\/strong><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t \t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Unifying increases the number of evaluated claims in SCORE from 100 to 250. (However, R1-R5 markets will only pay prizes for direct replications, because that’s what we said then.)<\/strong><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Sample Size & Statistical Power<\/h4>\n

A decent replication has to have a sample large enough that failure to detect a result is almost certainly due to the claim being wrong, rather than not looking hard enough. If we redo the study but with only 3 participants, the result is almost certainly noise.\u00a0<\/p>\n

We cannot just use the original sample size: most published studies are actually too small. Therefore we need to ensure sufficient power.<\/p>\n

You might want to read:<\/p>\n