{"id":930,"date":"2019-06-24T17:10:01","date_gmt":"2019-06-24T17:10:01","guid":{"rendered":"http:\/\/replicationmarkets.com\/?p=930"},"modified":"2019-09-04T09:09:32","modified_gmt":"2019-09-04T13:09:32","slug":"what-to-do","status":"publish","type":"post","link":"https:\/\/replicationmarkets.org\/index.php\/2019\/06\/24\/what-to-do\/","title":{"rendered":"What do I forecast?"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"930\" class=\"elementor elementor-930\">\n\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-a4dafbe elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"a4dafbe\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-53795d1\" data-id=\"53795d1\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-fb5570f elementor-widget elementor-widget-text-editor\" data-id=\"fb5570f\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p><span style=\"font-weight: 400;\">If you&#8217;ve just read\u00a0<\/span><a style=\"letter-spacing: 0px;\" href=\"https:\/\/replicationmarkets.com\/index.php\/2019\/06\/23\/what-is-high-powered-replication\/\" target=\"_blank\" rel=\"noopener\">What is a high-quality replication<\/a>, you know that our replications will be high-power &#8212; usually much higher than the original study &#8212; but not quite the &#8220;100 replications&#8221; ideal set out by DARPA:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2ef26f4 elementor-widget elementor-widget-blockquote\" data-id=\"2ef26f4\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"blockquote.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t \t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ce55fc8 elementor-widget elementor-widget-text-editor\" data-id=\"ce55fc8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>If you&#8217;re a forecaster, should this difference affect you, and if so, how? \u00a0One suggestion:<\/p><ul><li><strong>Big-picture person, in a hurry, or new?<\/strong>\u00a0Forecast the ideal. Maybe hedge a little to allow for noise.<\/li><li><strong>Detail person, experienced, have time<\/strong>? Maximize your accuracy. Especially in the market.<\/li><\/ul><p><span style=\"font-weight: 400;\">With high enough power, these are the same, but actually there will be some difference. \u00a0DARPA has asked us for Confidence Scores, but we can recover those from accurate forecasts &#8211; not necessarily the reverse. \u00a0This matters most in the market, because the market pays on actual ground truth: market incentives are closely tied to real accuracy. <\/span><\/p><p>But what should forecasters do? It depends.<\/p><ul><li>Either should do pretty well. Power is pretty high.<\/li><li>In the<strong> survey<\/strong>, there is a separate question for plausibility. Put your ideal Confidence Score there, and then adjust for the Replication question.<\/li><li>In the <strong>market<\/strong>, whichever suits you. A division of labor is efficient.<\/li><\/ul><p>The market is probably most accurate when forecasters specialize, because one kind of forecaster won&#8217;t have enough points. Two strategies are <em>ballparking<\/em> and <em>calibrating<\/em>.\u00a0<\/p><ul><li><strong>Ballparking<\/strong>: make early cheap edits to the ballpark of the ideal.\u00a0<\/li><li><strong>Calibrating<\/strong>: adjust the ballpark when appropriate.<\/li><\/ul><p><span style=\"font-weight: 400;\">Calibrators will want to get good at estimating the difference between ideal and actual, and maximizing predictive accuracy. \u00a0Some of their edits will be costly, so they need to know the LMSR scoring rule well, and get good at recovering points and other technical trading. \u00a0<\/span><\/p><p><span style=\"font-weight: 400;\">Ballparkers can be bigger-picture, but may need to get fast. They also have to watch their budget: smaller edits on more questions are likely more efficient. They need to know that the LMSR is a bowl-shaped function, getting more expensive as you push the ball (forecast) uphill to the extremes, and cheapest in reverse.\u00a0<\/span><\/p><p><span style=\"font-weight: 400;\">Both can prosper. The market pays for <em>information<\/em>: an early ballpark edit that moves the replication chance from 68% to 85% receives twice the reward of one that then moves it from 85% to 95%, assuming the claim replicates. \u00a0<\/span><\/p><p><span style=\"font-weight: 400;\">Participants in previous replication markets (70-80% accurate!) report doing well with two basic strategies:<\/span><\/p><ol><li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Estimate the <strong>plausibility<\/strong> of the claim itself. (A ballpark Confidence Score).<\/span><\/li><li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Attend to the <strong>statistics<\/strong> of the original study, chiefly p-value, power, and effect size. (Our starting prices now take some of this into account, so you have to work a little harder than before. More calibrating now.)<\/span><\/li><\/ol><p>Good luck! Be sure to follow the conversation and share your own thoughts.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-9566bc3 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"9566bc3\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-2446456\" data-id=\"2446456\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-ad9cee3 elementor-widget elementor-widget-text-editor\" data-id=\"ad9cee3\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h4>Follow Us<\/h4><p><span style=\"font-family: Montserrat, sans-serif; font-size: medium;\"><span style=\"white-space: pre-wrap;\">Although final results will not be available until mid-2020, you may follow the project to receive updates and relevant posts.<\/span><\/span><\/p><p><span style=\"font-size: 12pt; font-family: Montserrat, sans-serif; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;\"><strong>Replication Markets will be launched in August 2019 pending IRB approval; you can <a href=\"https:\/\/replicationmarkets.com\/index.php\/sign-up\/\" target=\"_blank\" rel=\"noopener\">sign up here<\/a> to be notified. <\/strong><\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>If you&#8217;ve just read&nbsp;What is a high-quality replication, you know that our replications will be high-power &#8212; usually much higher than the original study &#8212; but not quite the &#8220;100 replications&#8221; ideal set out by DARPA: Assume 100 replications of this study were performed, and a weighted average of their results (i.e., a meta-analysis) was&hellip;<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[20,19,14],"tags":[34,41,54,27,55,31],"class_list":["post-930","post","type-post","status-publish","format-standard","hentry","category-forecasting","category-prediction-markets","category-replication","tag-forecasters","tag-markets","tag-meta","tag-replication","tag-statistical-power","tag-survey"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/posts\/930","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/comments?post=930"}],"version-history":[{"count":13,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/posts\/930\/revisions"}],"predecessor-version":[{"id":2152,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/posts\/930\/revisions\/2152"}],"wp:attachment":[{"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/media?parent=930"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/categories?post=930"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/tags?post=930"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}