The Researcher's Tool Kit

Resources for working smarter

You call that significant?

Leave a comment

Echoing the sentiment of Robert Coe’s 2002 paper, It’s the Effect Size, Stupid, one of my former Stats professors always commented, “You call that significant?” when reported results did not include an effect size (regardless of the P value).  As Coe and many others have argued for years, reporting effect size with margin of error has many advantages over tests of statistical significance alone.cropped-cropped-toolkit-24157707-copy.jpg

Remember that the P value, which is reported as statistical significance, is a measure of the likelihood (or probability) that an observed difference between two groups is due to chance.  Keep in mind that the P value depends upon both effect size and sample size. Sometimes, a statistically significant result may mean only that a huge sample was used.  Statistical significance tells us only if there is a difference between two groups.  If reported in absence of effect size it does not provide adequate information for the consumer to completely  assess and understand the results.

So, how does effect size complete the story?  Effect size is a scale-free index that is independent of sample size.  It is a simple way to quantify the size of the differences between two groups.  Statistical significance tells us if the difference between two groups is likely due to chance and effect size tells us the magnitude of the difference.  Depending on the type of comparisons made, effect size  is estimated with different indices.  The two main indices used are (1) differences between the means of two groups and (2) measures of the associations or correlations between variables.  Many online effect size calculators  with explanations of their use for calculation are readily available.

Reporting effect sizes not only assists readers in understanding the magnitude of the differences reported but also contributes essential information to any field of study.  When you include effect sizes in results, you contribute in planning future studies as well as analyzing and reporting results from research studies. How so?  Remember that accurate effect size is needed when conducting:

  • sample size estimation
  • power analysis
  • meta-analysis

We’ll leave the final word about the “significance” of effect size with Jacob Cohen in “Things I have learned (so far) Am Psychol. 1990;45:1304–1312.”:

“The primary product of a research inquiry is one or more measures of effect size, not P values.”

Advertisements

Author: Jeanine Jesberg

Jeanine Jesberg is a grants consultant, Certified Research Administrator (CRA), and licensed clinical speech-language pathologist (CCC-SLP) specializing in work with academic institutions and non-profit organizations. Her multi-faceted career includes several positions, including Program Director, Director of Research Operations, and Executive Director, at the University of Chicago as well as Manager of Research Administration at Northwestern University with knowledge that spans strategic planning, conference planning, program management, budget development, research operations, and research administration. She also has over 15 years of experience as a speech-language pathologist in the roles of clinician, Clinical Instructor, Lecturer, private practice Founder/CEO, and speaker. Jeanine currently lives and works in Chicago, IL.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s