Our new preprint looks at how differences in decision-making by researchers influence the results they report. This paper had 100s of researchers independently analyse two datasets and report their findings. Surprisingly, this resulted in a nearly continuous set of reported effect sizes across a wide range!
What to do about this problem is less clear – maybe multiple analyses per dataset? Model averaging? Have multiple researchers analyse each dataset?
Each of these solutions is far from perfect and it will be interesting to see how the field reacts to this paper.