After a few posts praising ignorance I thought it was time to write about knowledge and power.
My economics professor quoted Richard Nixon as saying, “I’d give anything for a one-handed economist!” meaning economists always offer an added “on the other hand…” when talking policy. If you’ve ever talked with a statistician this might sound familiar, where any conceivable question you can ask is answered with two small words, “it depends.” You might think statisticians have only one absolute, to reject the null hypothesis when P is less than 0.05, whatever that means. At least this one thing seems to appease those hard-nosed statisticians. That’s the number that get’s research published, right? Well…it depends.
Leaving aside the philosophical and technical critiques of a p-value, there is something equally if not more important than statistical significance that every researcher must consider: is it practically significant? Today I built a model that predicted seed vigor with a high level of accuracy. All of the predictors were highly significant (I’m talking p<0.00001 significant!) including a new test which we hoped would help better manage inventory. I was thrilled, but the new test is costly and the powers that be wanted to know how much value it added to the predictive power of the model. Highly statistically significant wasn’t going to cut it. So I tested the classification error rates of two models, one including and one excluding the new test. Lo and behold the new test improved accuracy by a whopping 0.2%! Statistically significant! Break out the sparkling grape juice! Well…err…it depends of course. Was it practically significant? In this particular case 0.2% didn’t amount to a hill of beans but I’ve seen other situations where 0.2% paid for my salary 5 times over. So here’s the rub: be smart. Understand the process. Know what makes a difference in the real world. Ignorance is good when you’re exploring, but when it comes time to make decisions knowledge is power.