0.05

After a few posts praising ignorance I thought it was time to write about knowledge and power.

My economics professor quoted Richard Nixon as saying, “I’d give anything for a one-handed economist!” meaning economists always offer an added “on the other hand…” when talking policy.  If you’ve ever talked with a statistician this might sound familiar, where any conceivable question you can ask is answered with two small words, “it depends.”  You might think statisticians have only one absolute, to reject the null hypothesis when P is less than 0.05, whatever that means.  At least this one thing seems to appease those hard-nosed statisticians.  That’s the number that get’s research published, right?  Well…it depends.

Leaving aside the philosophical and technical critiques of a p-value, there is something equally if not more important than statistical significance that every researcher must consider:  is it practically significant? Today I built a model that predicted seed vigor with a high level of accuracy.  All of the predictors were highly significant (I’m talking p<0.00001 significant!) including a new test which we hoped would help better manage inventory.  I was thrilled, but the new test is costly and the powers that be wanted to know how much value it added to the predictive power of the model.  Highly statistically significant wasn’t going to cut it.  So I tested the classification error rates of two models, one including and one excluding the new test.  Lo and behold the new test improved accuracy by a whopping 0.2%!  Statistically significant!  Break out the sparkling grape juice!   Well…err…it depends of course.  Was it practically significant? In this particular case 0.2% didn’t amount to a hill of beans but I’ve seen other situations where 0.2% paid for my salary 5 times over.  So here’s the rub: be smart. Understand the process.  Know what makes a difference in the real world.  Ignorance is good when you’re exploring, but when it comes time to make decisions knowledge is power.

Advertisements

Leave a comment

Filed under Ordinary

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s