Effect sizes and power analysis

For a workshop on “Sample Size Justification” we are starting a discussion on here about the smallest effect size a researcher within empirical literary studies might be interested in.

As our field is not in the habit of reporting effect sizes and efforts to incorporate effect sizes and sample size justification in our research reports are just starting out, it is difficult to run a-priori power analyses to determine appropriate sample size for our studies.

Based on our reading of the pre-print paper by Daniël Lakens on “Sample Size Justification” (https://psyarxiv.com/9d3yf/), we want to start a discussion on which effect sizes are considered meaningful in our field. How do we determine whether an effect size is meaningful? Is it possible to to determine this even though our field investigates a great variety of highly subjective experiences?

In case the smallest effect size of interest can be agreed upon, it becomes possible to design a study with sufficient power and with a known Type II error rate.

After the workshop, the ELIT PhD’s will start the discussion here in this thread and anybody can feel free to add their two cents, and any relevant example studies that did report effect sizes, or even better: that reported the results of an a-priori power analysis.


This is a very important topic!
Especially in the context of Open Science, where the pre-registration of experiments is a recommended practice: in order to be able to properly plan a detailed experiment and analysis, we need to know the size of the population sample we need (and to do that we need some info about the expected effect size).

See: Preregistration (Center for Open Science)