{"id":5203,"date":"2020-11-01T21:21:28","date_gmt":"2020-11-02T02:21:28","guid":{"rendered":"http:\/\/replicationmarkets.com\/?p=5203"},"modified":"2020-11-02T08:01:09","modified_gmt":"2020-11-02T13:01:09","slug":"will-covid-19-studies-replicate","status":"publish","type":"post","link":"https:\/\/replicationmarkets.org\/index.php\/2020\/11\/01\/will-covid-19-studies-replicate\/","title":{"rendered":"Will COVID-19 Studies Replicate?"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"5203\" class=\"elementor elementor-5203\">\n\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-668de2f1 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"668de2f1\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-39fcf652\" data-id=\"39fcf652\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-5a526545 elementor-widget elementor-widget-text-editor\" data-id=\"5a526545\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\n<p class=\"wp-block-paragraph\">Replication Markets recently estimated replication probabilities for 100 COVID-19 studies from the social and behavioral sciences. We don\u2019t have all the results yet, but let\u2019s see what our forecasters thought. (Note: We just launched another COVID-19 project, we hope you\u2019ll <a href=\"https:\/\/covid19.replicationmarkets.com\/app\/register\">sign-up to participate<\/a> or <a href=\"https:\/\/replicationmarkets.com\/index.php\/rm-c19\/blog\/evaluating-covid-19-preprints\/\">learn more<\/a>).<\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><a>Reviewing our Previous COVID-19 Forecasts<\/a><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Big picture<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>~75% of studies (everything to the right of the blue line) are estimated to have a better than 50% chance of replicating successfully.<\/li><li>In prior research only ~50% of studies replicate successfully, so it\u2019s interesting to note that our forecasters are more optimistic about the studies in this sample.<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Comparing Surveys and Markets<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Participants in Replication Markets express their beliefs about the probability of successful replication two ways: with prediction surveys and markets. We\u2019ll compare the forecasts we get from each of these methods<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>The market estimates are slightly lower (by 1.9%)<\/li><li>The market estimates are more varied (ratio of variances = 1.5)<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Fig 1: Note that each point corresponds to the estimated replication probability for a different study.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img fetchpriority=\"high\" decoding=\"async\" width=\"700\" height=\"400\" data-recalc-dims=\"1\" src=\"https:\/\/i0.wp.com\/replicationmarkets.com\/wp-content\/uploads\/2020\/11\/Rplot01.png?w=1240&#038;ssl=1\" alt=\"\" class=\"wp-image-5204\" srcset=\"https:\/\/i0.wp.com\/replicationmarkets.org\/wp-content\/uploads\/2020\/11\/Rplot01.png?w=700&amp;ssl=1 700w, https:\/\/i0.wp.com\/replicationmarkets.org\/wp-content\/uploads\/2020\/11\/Rplot01.png?resize=300%2C171&amp;ssl=1 300w\" sizes=\"(max-width: 700px) 100vw, 700px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<h4 class=\"wp-block-heading\"><a>Do surveys and markets agree about individual studies?<\/a><\/h4>\n\n\n\n<ul class=\"wp-block-list\"><li>The plot below indicates a fair amount of agreement on replication probabilities between the surveys and the market.<\/li><li>The points above the line are studies for which the survey was more optimistic than the market.<\/li><li>We hope combining surveys and markets can improve the accuracy of our forecasts<\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\"><a>Are these forecasts accurate?<\/a><\/h4>\n\n\n\n<p class=\"wp-block-paragraph\">We evaluate our replication forecasts with replication results. As of today we have only 8 replication results. Of the 100 we forecast, we expect 20-50 to eventually receive a replication efforts (as part of our research program, DARPA SCORE).<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>All 8 studies replicated successfully, a positive surprise!<\/li><li>We assigned &gt; 0.6 probability of replication success to each successful replication.<\/li><li>Despite being 8 for 8, we are certain our forecasts are imperfect. The 8 studies replicated look \u201cmore obvious\u201d than most. We will learn more about where we were right and wrong over time as more replication results arrive.<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"700\" height=\"550\" data-recalc-dims=\"1\" src=\"https:\/\/i0.wp.com\/replicationmarkets.com\/wp-content\/uploads\/2020\/11\/market_vs_survey_forecasts_scatterplot.png?w=1240&#038;ssl=1\" alt=\"\" class=\"wp-image-5205\" srcset=\"https:\/\/i0.wp.com\/replicationmarkets.org\/wp-content\/uploads\/2020\/11\/market_vs_survey_forecasts_scatterplot.png?w=700&amp;ssl=1 700w, https:\/\/i0.wp.com\/replicationmarkets.org\/wp-content\/uploads\/2020\/11\/market_vs_survey_forecasts_scatterplot.png?resize=300%2C236&amp;ssl=1 300w\" sizes=\"(max-width: 700px) 100vw, 700px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you find this interesting, please <a href=\"https:\/\/covid19.replicationmarkets.com\/app\/register\">sign-up to participate<\/a> or <a href=\"https:\/\/replicationmarkets.com\/index.php\/rm-c19\/blog\/evaluating-covid-19-preprints\/\">learn more<\/a>.<\/p>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4c4a8f7 elementor-widget elementor-widget-text-editor\" data-id=\"4c4a8f7\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Note: Cross-posted on RM&#8217;s COVID-19 project blog [<a href=\"https:\/\/replicationmarkets.com\/index.php\/rm-c19\/blog\/will-covid-19-studies-replicate\/\">LINK<\/a>]<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Replication Markets recently estimated replication probabilities for 100 COVID-19 studies from the social and behavioral sciences. We don\u2019t have all the results yet, but let\u2019s see what our forecasters thought. (Note: We just launched another COVID-19 project, we hope you\u2019ll sign-up to participate or learn more). Reviewing our Previous COVID-19 Forecasts Big picture ~75% of&hellip;<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-5203","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/posts\/5203","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/comments?post=5203"}],"version-history":[{"count":2,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/posts\/5203\/revisions"}],"predecessor-version":[{"id":5222,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/posts\/5203\/revisions\/5222"}],"wp:attachment":[{"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/media?parent=5203"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/categories?post=5203"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/tags?post=5203"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}