{"id":4067,"date":"2020-06-12T08:02:00","date_gmt":"2020-06-12T12:02:00","guid":{"rendered":"http:\/\/replicationmarkets.com\/?p=4067"},"modified":"2020-06-15T10:30:20","modified_gmt":"2020-06-15T14:30:20","slug":"a-brief-tour-of-leaderboards","status":"publish","type":"post","link":"https:\/\/replicationmarkets.org\/index.php\/2020\/06\/12\/a-brief-tour-of-leaderboards\/","title":{"rendered":"A brief tour of leaderboards"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"4067\" class=\"elementor elementor-4067\">\n\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-89a86f5 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"89a86f5\" data-element_type=\"section\" data-e-type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-5bb24c7\" data-id=\"5bb24c7\" data-element_type=\"column\" data-e-type=\"column\">\n\t\t\t<div class=\"elementor-widget-wrap elementor-element-populated\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-66b962d elementor-widget elementor-widget-text-editor\" data-id=\"66b962d\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h3>The shorter version:<\/h3><p><strong>Market leaderboards<\/strong> show the <em>expected value<\/em> of your Round portfolio, in points (\u0e3f). They refresh every few hours. <em>If <\/em>the market is accurate, they will roughly predict prizes.<\/p><p><strong>Survey leaderboards<\/strong> sum your <em>rank score<\/em> for each batch in that Round. Rank score is just inverted rank: 1 is the worst. Survey <em>prizes <\/em>go to the top 4 per batch, so will roughly correlate.\u00a0 (See <u>this blog post<\/u> for details, including a revision that did not affect prizes.)<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9e0310a elementor-widget-divider--view-line elementor-widget elementor-widget-divider\" data-id=\"9e0310a\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"divider.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"elementor-divider\">\n\t\t\t<span class=\"elementor-divider-separator\">\n\t\t\t\t\t\t<\/span>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-93d45c8 elementor-widget elementor-widget-text-editor\" data-id=\"93d45c8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h3>Survey Leaderboards: longer version:<\/h3>\n<p>While we won\u2019t <em>know<\/em> your accuracy until the replication results are announced, we do our best to <em>estimate<\/em> it and keep score anyway. The survey leaderboards reward both quality (estimated accuracy) and participation (surveys completed). We provide them for motivation &amp; feedback:<\/p>\n<ul>\n<li>We want to recognize and reward the most accurate forecasters.<\/li>\n<li>We hope even this limited feedback gives you some idea of whether you\u2019re on the right track.<\/li>\n<\/ul>\n<p><strong>How does RM score surveys for the leaderboard?<\/strong><\/p>\n<ul>\n<li>Estimate the accuracy of every forecasting survey using peer prediction. (See <a href=\"https:\/\/replicationmarkets.com\/index.php\/2020\/01\/02\/robustness-of-monthly-prizes\/\">this post<\/a> for more info.) &nbsp;Rank these.<\/li>\n<li>Each completed batch earns 1..N points based on your rank:\n<ol>\n<li>Least accurate gets 1 point, plus one point per rank, so\u2026<\/li>\n<li>The most accurate receives N points, the number who completed that batch.<\/li>\n<\/ol>\n<\/li>\n<li>Your total round leaderboard score <em>sums<\/em> the batch scores for that round.<\/li>\n<\/ul>\n<p>As participation increases, doing well becomes more important. In Round 8, winning a single batch earned ~35 leaderboard points.<\/p>\n<p><strong>How does RM estimate survey forecasting accuracy before replication results are known?<\/strong><\/p>\n<p>We are excited to be using new methods in peer prediction. We believe they are fair and accurate. Though we don\u2019t we don\u2019t believe they are vulnerable to manipulation<a href=\"#_ftn1\" name=\"_ftnref1\">[1]<\/a>, we decided to remain vague on their details for now. Our methods will eventually be published publicly.<\/p>\n<p><strong>The leaderboard needn\u2019t match the prize winners list perfectly<\/strong><\/p>\n<p>Completing 30 batches of surveys and earning a respectable 5<sup>th<\/sup> place in each would not win a single cash prize, but it would place you fairly high on the leaderboard. Much higher than a person who completed one batch and won the $80 first place prize. The people who win the most money obviously complete a lot of batches and score in the top four in many of them.<\/p>\n<p><strong>Changes to the leaderboard calculations<\/strong><\/p>\n<p>Note that in Rounds 1-5 we initially translated rank into points slightly differently\u2026 If the most popular batch had 35 participants, then every batch in the round gave 35 points to the 1<sup>st<\/sup> place winner. It also wasn\u2019t required that participants complete every survey in the batch in order to win some points. We started assigning points differently in Round 6 to better align with how we were choosing prizes, and because it seemed fair that 10<sup>th<\/sup> place, out of 10, should earn fewer points than 10<sup>th<\/sup> place out of 35. On June 8, 2020 we adjusted the scores for rounds 1-5 to use the same scoring formula. This had a negligible effect on the ranking of participants that completed more than 5 batches of surveys. Prize allocation is unchanged.<\/p>\n<p><a href=\"#_ftnref1\" name=\"_ftn1\"><\/a><\/p>\n<hr>\n<p><a href=\"#_ftnref1\" name=\"_ftn1\">[1]<\/a> That is, we think it\u2019s easier for you to coordinate on the truth than to game the method. See <a href=\"https:\/\/replicationmarkets.com\/index.php\/2020\/01\/02\/robustness-of-monthly-prizes\/\">this post<\/a> for some early robustness checks, or search the literature for &#8220;peer prediction&#8221; and &#8220;surrogate scoring&#8221;.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>The shorter version: Market leaderboards show the expected value of your Round portfolio, in points (\u0e3f). They refresh every few hours. If the market is accurate, they will roughly predict prizes. Survey leaderboards sum your rank score for each batch in that Round. Rank score is just inverted rank: 1 is the worst. Survey prizes&hellip;<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[1],"tags":[],"class_list":["post-4067","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/posts\/4067","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/comments?post=4067"}],"version-history":[{"count":5,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/posts\/4067\/revisions"}],"predecessor-version":[{"id":4079,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/posts\/4067\/revisions\/4079"}],"wp:attachment":[{"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/media?parent=4067"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/categories?post=4067"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/replicationmarkets.org\/index.php\/wp-json\/wp\/v2\/tags?post=4067"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}