{"pk":27015,"title":"The statistical significance filter leads to overconfident expectations of replicability","subtitle":null,"abstract":"We show that publishing results using the statistical signif-icance filter—publishing only when the p-value is less than0.05—leads to a vicious cycle of overoptimistic expectationof the replicability of results. First, we show analytically thatwhen true statistical power is relatively low, computing powerbased on statistically significant results will lead to overesti-mates of power. Then, we present a case study using 10 exper-imental comparisons drawn from a recently published meta-analysis in psycholinguistics (J ̈ager et al., 2017). We show thatthe statistically significant results yield an illusion of replica-bility. This illusion holds even if the researcher doesn’t con-duct any formal power analysis but just uses statistical signifi-cance to informally assess robustness (i.e., replicability) of re-sults.","language":"eng","license":{"name":"","short_name":"","text":null,"url":""},"keywords":[{"word":"Statistical significance; p-values; replicability"}],"section":"Talks: Papers","is_remote":true,"remote_url":"https://escholarship.org/uc/item/2mc3x8dc","frozenauthors":[{"first_name":"Shravan","middle_name":"","last_name":"Vasishth","name_suffix":"","institution":"University of Potsdam,","department":""},{"first_name":"Andrew","middle_name":"","last_name":"Gelman","name_suffix":"","institution":"Columbia University","department":""}],"date_submitted":null,"date_accepted":null,"date_published":"2017-01-01T13:00:00-05:00","render_galley":null,"galleys":[{"label":"PDF","type":"pdf","path":"https://journalpub.escholarship.org/cognitivesciencesociety/article/27015/galley/16651/download/"}]}